Showing posts with label binary. Show all posts
Showing posts with label binary. Show all posts

Sunday, November 7, 2010

Select Odd Rows in MySQL

Any time you can use a bitwise operator in place of a function call, it's a good thing. Selecting odd rows from a table in MySQL that have an auto_increment primary key is no exception.

SELECT * FROM mytable WHERE id & 1;


If you're unfamiliar with a bitwise AND, basically what it says is return true of the left operand has the same bits set as the right operand. The number 1 only has one bit set, and conveniently enough that one bit is the same bit that will only be present in odd numbers.

Friday, September 5, 2008

Web File Formats 101

These are file extensions or formats you're likely to come across at one point or another if you're going to get into designing web sites. They're split into common groups, include a brief description of where you're likely to see or use them, and clicking the *.EXTENSION link in the titles will take you to a Wikipedia entry with a lot more details on that file format.

This list is a collection of about 45 file formats I've used enough times to remember when presented with a huge list of file formats after roughly five years of experience.




Browser File Formats

Browser file formats are files that instruct web browsers to do something directly. Many technologies may work with browser file formats, but browsers are generally the things that make use of them.

*.AS Adobe/Macromedia Action script file

AS files are generally used as library or optional include files in Flash or Flex applications.

*.CSS Cascading Style Sheet file

CSS files contain CSS rules which web browsers use to decide what web pages should look like.

*.HTM / *.HTML Hyper Text Markup Language

HTML files are web pages. HTML is what files such as PHP, ASP, CFML, and JSP produce when they're executed.

*.ICO Icon file

ICO files are commonly used as Favicons (that little icon in the address bar)

*.JS Javascript file

JS files are used to store Javascript. Javascript is a cornerstone in many of the applications on the web today. Javascript provides a way for web developers to utilize the browsers rendering engine and make changes to the content of a web page without needing to make a trip back to the web server to generate a new page with the updated content. Javascript is becoming more popular for producing animation in web pages since the jQuery, Prototype, and Moo-Tools Javascript libraries were developed.

*.SWF Adobe/Macromedia Shockwave Flash file

SWF files are generally used in Rich Internet Applications and require the Adobe/Macromedia Flash Player to view.

*.VBS Microsoft Visual Basic Script file

VBS files are similar in concept to JS files but they often have control over operation system functionality normally off limits to JS, however VBS files are only supported by Microsoft Internet Explorer versus JS which is available on almost every popular web browser. VBS is generally only found on private Microsoft Windows based Intranet sites, though even there it's being replaced by newer offerings from Microsoft.

Image File Formats

Image file formats are files used to store and transfer photographs, vector art, logos, favicons and things like that. They're also used to store information artists use to modify the images in image manipulation applications such as Photoshop and Fireworks.

*.BMP Bitmap Graphic file

BMP files are an early image format with little or no compression applied. BMP files are sometimes found on websites, but have generally been replaced by formats such as PNG, GIF, and JPG in recent years.

*.GIF Graphic Interchange Format file

GIF files are image files that sometimes include simple frame-based animation and alpha transparency. PNG has gained popularity over GIF in recent years do the the superior color quality of PNG, however transparency problems with Microsofts Internet Explorer 6 have slowed the transition from GIF to PNG.

*.JPG / *.JPEG Joint Photographic Group image file

JPG files are image files designed specifically for photos. The compression method used has been optimized for working with complicated images, making it especially well suited for transferring photos on the web.

*.PNG Portable Network Graphic file

PNG is an image file format that has gained popularity over GIF in recent years do the the superior color quality of PNG, however transparency problems with Microsofts Internet Explorer 6 have slowed the transition from GIF to PNG. PNG is also the native file format used by Adobe/Macromedia Fireworks, similar to how PSD is the native file format used by Adobe Photoshop. PNG has the ability to store information about layers when used in Fireworks in much the same way that PSD can store information about layers in Photoshop.

*.PSD Photoshop Document file

PSD is an image file format and is the native format used by Adobe Photoshop. PSD allows image authors to store information about layers in an image when edited in Photoshop in much the same way that PNG enables layer information when edited in Fireworks. Unlike PNG which generally has multiple export modes and can be exported for use on the web, PSD is primarily used for raw or drafts before exporting the finished pieces of the image to other image formats such as PNG, GIF, or JPG.

*.SVG Scalable Vector Graphics file

SVG files are vector image files. Unlike common raster image formats that store information about each pixel in the image, vector image formats only store information about how to draw the image, which generally consists of coordinates to place simple shapes and how to fill them with color.

Audio/Video File Formats

Audio/Video file formats are used to store and transfer audio and video.

*.AVI Audio Video Interleave file

AVI files are commonly used to distribute video.

*.FLV Flash Video file

FLV files are usually not observed directly, but rather played through an SWF file. FLV files are used for distributing streaming video.

*.MOV / *.QT Apple QuickTime Video Clip file

MOV or QT files are used to distribute video, similar to AVI files. Quicktime is generally known for a great level of quality in video.

*.MP3 MPEG Audio Stream, Layer III file

MP3 files are used to distribute audio. MP3 files use a compressed version of the audio information and are generally many times smaller in size compared to WAV files, despite including extra information about the audio such as the author, title, album name, and description.

*.MPEG / *.MPG MPEG 1 System Stream file

MPEG or MPG files are used to distribute video, similar to AVI, MOV, and QT files.

*.WAV Waveform Audio file

WAV files are an early audio format file which has all but been replaced by MP3 on the web due to superior compression and information embedding provided by the MP3 audio file format.

*.WMA Microsoft Windows Media Audio file

WMA is an audio file similar to MP3 and WAV developed by Microsoft. Like WAV, WMA has been replaced to a large extend on the web by MP3.

Compression File Formats

Compression file formats are used to apply compression to single, or groups of files in order to reduce the size needed to store and transfer the files, as well as making large groups of files easier to transfer at once.

*.BZ2 BZIP2 compressed archive file

BZ2 files are generally used by programmers to distribute compressed source code packages.

*.GZ GZIP compressed archive

GZ files are sometimes used in the same way BZ2 files are, for programmers to distribute source code packages. Some systems use GZIP compression on their old log files to save space which results in file names such as "date.log.gz". Some web servers use GZIP compression to compress data before sending it from a server to a browser.

*.RAR winRAR compressed archive

RAR files are often used by individuals to compress and transfer large files, or large groups of files. Similar to GZ, BZ2, and ZIP files.

*.ZIP Compressed archive file

ZIP files are similar to RAR, BZ2, and GZ files however ZIP files are generally a lot more common. ZIP files package and to a point compress other files to be transferred from system to system.

Misc File Formats

Misc file formats are generally not designed specifically for web sites, but have found numerous uses on them.

*.*.BAK Backup copy of file

Backup. The second asterisk is generally the original file extension.Rather than trying to open this file, you will usually rename it and remove the .BAK from the end, then open it with the appropriate application

*.DOC Microsoft Word document

DOC files are often used by companies to distribute bundled information suitable for printing. DOC files are a favorite of companies for sending contracts via e-mail. DOC files are similar in concept to PDF files.

*.INI Information/configuration file

INI files are commonly used to store persistent configuration data for web servers and web browsers. INI files are also used in the same way for applications that have nothing to do with web design. Using XML files to store configuration data instead of INI files has been gaining popularity in recent years.

*.KML Google Earth/Keyhole place mark file

KML files are used to share specific locations, or placeholders in the popular Google Earth application. KML files are becoming increasingly popular on business web sites who want to provide directions for customers.

*.PDF Acrobat Portable Document Format file

PDF files are often used by companies to distribute bundled information suitable for printing, similar to DOC files. Some print shops will ask for PDF files when you need to have fliers or brochures printed out. Some companies will make PDF copies of important information found on their web site available for people do download and read offline, or print out and read away from their desk.

*.RDF Resource Description Framework file

RDF files are often used to deliver RSS content. RDF RSS feeds are generally more specialized in comparison to the generic XML versions of RSS files.

*.TXT Plain text file

TXT files are plain text files. TXT files generally display the characters as-is with no consideration for special style codes.

*.XML Extensible Markup Language file

XML files are generally used to store structured information. Many web sites use AJAX to load XML in the background and update small portions of web pages without needing the server to generate the entire web page over again with slightly changed information. XML files are also gaining popularity as a settings or preference file format. The generic yet descriptive format used to section data off makes XML the perfect file format for storing raw information that will be displayed in many different ways.

Server File Formats

Server file formats are generally scripts that web servers execute in order to produce web pages dynamically. these types of files are also often used to perform maintenance on the server through automation.

*.ASP Microsoft Active Server Pages

ASP files are an earlier version of ASPX files. ASP files are similar in concept to CFML, PHP, ASPX, RB, and JSP files.

*.ASPX Microsoft ASP.NET source code file

ASPX files are a more recent version of ASP files. ASPX files are used on web servers to prepare HTML documents to send to a web browser. ASPX files are similar in concept to ASP, CFML, PHP, RB, and JSP files.

*.CFM / *.CFML Coldfusion Markup Language file

CFML files are used on web servers to prepare HTML documents to send to a web browser. CFML files are similar in concept to ASPX, PHP, ASP, RB, and JSP files.

*.CGI Common Gateway Interface file

CGI files are generally used to provide a bridge, or gateway, between a web server and applications installed on the server such as Python, Perl, sed, and many others.

*.JSP Java Server Pages

JSP files are used on web servers to prepare HTML documents to send to a web browser. JSP files are similar in concept to ASPX, PHP, ASP, RB, and CFML files.

*.LOG Log file

LOG files are generally exactly what the name implies, a log. Web servers use them to store details about each visit to a web site as well as to store information useful to administrators and developers in the event of a problem. Old, or archived LOG files are often compressed using GZIP compression, which results in "filename.log.gz" types of file names.

*.MDB Microsoft Access Database file

MDB files are a Microsoft database data storage format. The MySQL database engine has been gaining popularity over MDB in recent years. There are many MDB to MySQL converters and conversion methods available on-line.

I posted a how to for Importing MDB to MySQL on a Debian server not too long ago.

*.PHP Hypertext PreProcessor script file

PHP files are used on web servers to prepare HTML documents to send to a web browser. PHP files are similar in concept to ASPX, JSP, RB, ASP, and CFML files. PHP is one of the most popular pre-processors used on the web today.

*.PL Perl script file

PL files are Perl script files. Perl is one of the first scripting languages to be used on the web. Perl is often used in conjunction with CGI. In recent years PHP has stolen much of Perls thunder, leaving Perl used primarily by veterans.

*.RB Ruby script / Ruby on Rails class file

RB files are used on web servers to prepare HTML documents to send to a web browser. RB files are similar in concept to ASPX, JSP, PHP, and CFML files.

*.SHTML / *.SSI HTML including Server Side Includes

SHTML files are used on web servers to tell the web server that there's lines in the file that need to be replaced with the contents of other files before sending the contents of the file to the web browser. SHTML is generally a very simple version of what file types such as ASP, ASPX, CFML, JSP, PHP, and RB are designed for.

*.SQL Structured Query Language data file

SQL files generally contain the description of a database and the database table layouts. In some cases SQL files will contain information dumps from database tables, though this is generally reserved for small databases.

.htaccess Apache dynamic configuration file

.htaccess files are commonly used to make runtime adjustments to an Apache web server configuration without needing to restart the Apache server. Not all configuration options are able to be altered via .htaccess and depending on the servers configuration at start up there might not be any options that can be configured via .htaccess

Friday, July 18, 2008

Byte Chart w/ Large Numbers

Chart showing filesizes

It took me entirely too long to lookup how many digits were in a Petabyte, so after some hunting I put together this neatly organized chart displaying up to the 25th digit or 1024^8th place.
I tried to arrange the data in such a way that the chart shouldn't need much explaination or any legends or keys.

It shows
  • kibibyte / kilobyte filesize
  • mebibyte / megabyte filesize
  • gibibyte / gigabyte filesize
  • tebibyte / terabyte filesize
  • pebibyte / petabyte filesize
  • exbibyte / exabyte filesize
  • zebibyte / zettabyte filesize
  • yobibyte / yottabyte filesize


I noticed an easy way to remember which digit a Yottabyte starts with, Y is the 25th letter in the English alphabet & 25 is the digit a Yottabyte starts at. The same goes for P and Petabyte.

Thursday, December 6, 2007

BOM Squad

Cartoony Bomb
Since posting about phpBB3 and UTF-8 BOM/Signature in October, I discovered there seems to be alot of people tripping over UTF-8 encoded files that have a BOM, otherwise known as a Signature embeded.
Most people seem to know what the BOM is and what it does, but don't know how to get rid of it or how it got there.

Well, now seems like a good time to go over how the BOM gets in files, how to find out if an editor is silently prepending a BOM to UTF-8 encoded files, and how it can be removed if needed.

A major source of these BOM headaches seems to be Notepad on Windows. When saving a file with UTF-8 encoding using Notepad on Windows, it automaticly prepends the BOM to the file but doesn't inform the user about this.

I think this is partly due to quite a few people suggesting that Notepad can be used to easily and quickly convert files to UTF-8 in order to solve issues with strange characters showing up in files.

utf-8 Windows Notepad
Here's a screenshot of the choices you get when saving a file using Notepad. You can see there's no mention of BOM, or Signature there.
I can't help but wonder why Microsoft decided to diferentiate between big & little endian for Unicode but not leave a choice for the UTF-8 BOM.

From what I understand Windows text editors are the main contributers of UTF-8 BOM in files. UNIX type system applications generally don't include the BOM because it can cause problems with configuration files, which are primarily text files.

The first thing to do is check your editors' settings, particularly areas having to do with encoding, very well for UTF-8, BOM, and or Signature. You may have controll over what your editor does and it's good to be aware if you do.

One method to determine if your editor is secretly prepending the BOM to your files is to save an empty file using UTF-8 encoding and look at the filesize. If what should be an empty file has a filesize of three(3), your editor is prepending a BOM. You might want to consider leaving a single character in the file & making sure it's not four(4) bytes instead, some editors may default to non-UTF encoding for empty files.

I use a Notepad replacement called Notepad2 for quick editing. That application gives me an option to save UTF-8 encoded files with or without the BOM, though it calls it a Signature. Notepad2 displays which encoding is used in the file on the applications status bar which comes in handy from time to time. The application is self-contained and compressed is only about 250KB, easily carried around on a USB memory stick.

Another option for getting rid of the BOM is using the Perl script in this forum thread. I tried it myself (as seen in that thread), it works exactly as expected. This is probably the one of the easiest options if the files are on a Ubuntu or Linux system.

I'm sure there's plenty of people looking for a PHP solution to removing the BOM, so here's function designed for that situation called debom_utf8. It should work in either PHP4, PHP5. The methods used are generic enough that it should continue to work in PHP6.

<?php
/*
@author: http://develobert.blogspot.com/
@description : PHP Function to remove UTF-8 BOM/Signature from the beginning of a file.
@param $filename: Name of the file a BOM should be looked for and removed from.
@returns (bool): Returns true if the file didn't, or no longer contain(s) a BOM, false on error.

@example usage: echo debom_utf8('BOM.txt') ? 'free of BOM' : 'error';
*/
function debom_utf8($filename = '')
{
if($size = filesize($filename) && $size < 3)
{// BOM not possible
return true;
}
if($fh = fopen($filename, 'r+b'))
{
if(bin2hex(fread($fh, 3)) == 'efbbbf')
{
if($size == 3 && ftruncate($fh, 0))
{// Empty other than BOM
fclose($fh);
return true;
}
else if($buffer = fread($fh, $size))
{// Shift file contents to beginning of file
if(ftruncate($fh, strlen($buffer)) && rewind($fh))
{
if(fwrite($fh, $buffer))
{
fclose($fh);
return true;
}
}
}
}
else
{// No BOM found
fclose($fh);
return true;
}
}
return false;
}
?>


Comments on other methods to deal with BOM disposal are more than welcome.

Sunday, October 14, 2007

phpBB3 & UTF-8 BOM/Signature

Update: (12-6-2006) Another post about UTF-8 BOM/Signature

When phpBB.com says language files need to be saved as UTF-8 with no BOM, they have a good reason. This also goes for other PHP files as well.

Apparently PHP has a bug which causes it to treat the BOM marker in a UTF-8 encoded PHP file as data which should be sent immediately, instead of throwing it away.
This prevents PHP from sending headers via the header() method, I'm guessing it can cause some havok with cookies as well.

A tell tale sign of this is PHP error messages that look similar to the following.

With phpBB3, the error message will look somthing like this
[phpBB Debug] PHP Notice: in file /includes/functions.php on line 3372: Cannot modify header information - headers already sent by (output started at /geoip_install.php:1)


PHP files in general will have a similar error message with slightly different wording, note the character position pointer at the end of the filename being "1".
Warning: Cannot modify header information - headers already sent by (output started at /var/www-partition/phpBB3/geoip_install.php:1) in /var/www-partition/phpBB3/includes/functions.php on line 3368


Now, in case your editor causes you to become confused because of the wording in the encoding options being somthing along the lines of Unicode BOM and UTF-8 Signature, it is UTF-8 Signature that should be considered UTF-8 BOM.
I'm not sure how many editors call it a Signature instead of a BOM, I know Notepad2 does though.