How to mount/format a USB mass storage device (pen drive) in Linux

Using a USB mass storage device with Linux is not difficult at all. Linux is well equipped with command line utilities which can help accomplish this task in few easy steps.

Step1: Determine device file

When a USB device is connected to a machine running Linux, Linux kernel (brain of Operating System) will detect that a device has been connected to it and will log a message like this:

usb.c: registered new driver usb-storage
scsi3 : SCSI emulation for USB Mass Storage devices
Vendor: Generic Model: USB Disk Rev: 1.10
Type: Direct-Access ANSI SCSI revision: 02
Attached scsi removable disk sdb at scsi3, channel 0, id 0, lun 0
SCSI device sdb: 7897088 512-byte hdwr sectors (4043 MB)
sdb: Write Protect is off
sdb: sdb1 sdb2 sdb3 sdb4
WARNING: USB Mass Storage data integrity not assured
USB Mass Storage device found at 2
USB Mass Storage support registered.

The message above is obtained by running command
dmesg

dmesg command display message logged by Linux kernel. The message displayed above can be found at the end of log for a device connected a while ago. You can also run dmesg|tail to see last few lines.

The message above says that a USB device has been detected and the device file (don't bother what a device file is) is sdb for the whole device and 4 partitions are available at sdb1 sdb2 sdb3 sdb4. The device file are usually located under /dev directory. So
/dev/sdb is the device file for whole USB device and partitions are
/dev/sdb1
/dev/sdb2
/dev/sdb3
/dev/sdb4

Skip Step2 and Step3 if you don't want to change partitions and reformat partitions.

Step2: Alter partition table

fdisk utility helps in add/delete partitions. In our example, we have 4 partitions, lets delete them and create 1 partition of type FAT32.

Run fdisk as root user or sudo

1. Type sudo fdisk /dev/sdb to run fdisk program. It will display a command line interface.

#sudo fdisk /dev/sdb

Command (m for help): m
Command action
a toggle a bootable flag
b edit bsd disklabel
c toggle the dos compatibility flag
d delete a partition
l list known partition types
m print this menu
n add a new partition
o create a new empty DOS partition table
p print the partition table
q quit without saving changes
s create a new empty Sun disklabel
t change a partition's system id
u change display/entry units
v verify the partition table
w write table to disk and exit
x extra functionality (experts only)


2. Type p to print partition table

Command (m for help): p

Disk /dev/sdb: 4043 MB, 4043309056 bytes
125 heads, 62 sectors/track, 1018 cylinders
Units = cylinders of 7750 * 512 = 3968000 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 ? 100405 247697 570754815+ 72 Unknown
Partition 1 has different physical/logical beginnings (non-Linux?):
phys=(357, 116, 40) logical=(100404, 79, 11)
Partition 1 has different physical/logical endings:
phys=(357, 32, 45) logical=(247696, 24, 51)
Partition 1 does not end on cylinder boundary.
/dev/sdb2 ? 21767 271577 968014120 65 Novell Netware 386
Partition 2 has different physical/logical beginnings (non-Linux?):
phys=(288, 115, 43) logical=(21766, 48, 47)
Partition 2 has different physical/logical endings:
phys=(367, 114, 50) logical=(271576, 60, 42)
Partition 2 does not end on cylinder boundary.
/dev/sdb3 ? 241276 491086 968014096 79 Unknown
Partition 3 has different physical/logical beginnings (non-Linux?):
phys=(366, 32, 33) logical=(241275, 3, 30)
Partition 3 has different physical/logical endings:
phys=(357, 32, 43) logical=(491085, 14, 39)
Partition 3 does not end on cylinder boundary.
/dev/sdb4 ? 372346 372354 27749+ d Unknown
Partition 4 has different physical/logical beginnings (non-Linux?):
phys=(372, 97, 50) logical=(372345, 119, 25)
Partition 4 has different physical/logical endings:
phys=(0, 10, 0) logical=(372353, 14, 33)
Partition 4 does not end on cylinder boundary.

Partition table entries are not in disk order


3. Type d then 1 to delete first partition.

Command (m for help): d
Partition number (1-4): 1


4. Repeat command d to delete other partitions.

5. See current partition table with command p

Command (m for help): p

Disk /dev/sdb: 4043 MB, 4043309056 bytes
125 heads, 62 sectors/track, 1018 cylinders
Units = cylinders of 7750 * 512 = 3968000 bytes

Device Boot Start End Blocks Id System


Note: These changes are still not saved to the device. So if you want to quit, exit fdisk using command q

6. Create new partition with command n and then enter p for primary e for extended partition. If you want to have less than 5 partitions in your device, then enter p and create a primary partition.

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1018, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1018, default 1018):
Using default value 1018

Command (m for help): p

Disk /dev/sdb: 4043 MB, 4043309056 bytes
125 heads, 62 sectors/track, 1018 cylinders
Units = cylinders of 7750 * 512 = 3968000 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 1018 3944719 83 Linux


7. The new partition is type Linux. If you want to create partition of type say "FAT32" then use t command to toggle partition's ID.

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): b
Changed system type of partition 1 to b (Win95 FAT32)


8. See you changes and if fine with it, then save changes and exit fdisk.

Command (m for help): p

Disk /dev/sdb: 4043 MB, 4043309056 bytes
125 heads, 62 sectors/track, 1018 cylinders
Units = cylinders of 7750 * 512 = 3968000 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 1018 3944719 b Win95 FAT32

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: If you have created or modified any DOS 6.x
partitions, please see the fdisk manual page for additional
information.
Syncing disks.


Step3: Format partition

If you have create new partition(s), then fdisk command just create/alters partition table. The partition still need to be formatted to make it usable.

Command mkfs helps in formatting a partition. For example:

#sudo mkfs -t vfat /dev/sdb1
mkfs.vfat 2.8 (28 Feb 2001)

Step4: Mount a partition

When we connect a device to Windows, it mounts it automatically as some drive such as F:. In linux, we need to do it manually. Lets say we want to mount /dev/sdb1 (First partition) at /mnt/d. So create this directory
#sudo mkdir /mnt/d

and run command
#sudo mount -t vfat /dev/sdb1 /mnt/d to mount the partition.

Finally, We have mounted the USB device and its available for use under directory /mnt/d.


Step5: Unmount partition

To disconnect device, we have to unmount all the partitions previously mounted. This can be done as follows:
#sudo umount /dev/sdb1
or
#sudo eject /dev/sdb

With these easy steps, we can easily use USB mass storage devices with Linux.

Useful packages for Linux

I like to install some useful stuff on my linux machine.
To make my workstation more productive. :)

Here is the list of packages I usually install:

- screen
- freenx.i386

I use yum package manager on Fedora.

How to convert an integer to little endian or big endian

As discussed in my previous post "Little endian vs Big endian", there are some scenario where we need to take care of endianness of system. Consider an example where you want to send data over network to some remote systems. The systems can be of any endianness. All interested parties decide that the data transmitted over internet will be in big endian format. The system receiving data will then convert it to local endian format.

Let's write function to write and read bytes from a 64 bit integer. I am considering 64 bit integer because an endian related C program written with int may work fine on 32 bit platform but will fail on 64 bit platform. So, we will be using strict data types. More on C data types related issues here.

1. Extract bytes in big endian sequence from uint64_t irrespective of the execution platform.

void
uint64ToByteArray (uint64_t num, size_t bytes, unsigned char *arr)
{
size_t i;
unsigned char ch;
for (i = 0; i < bytes; i  )
 {
   ch = (num >> ((i & 7) << 3)) & 0xFF;
   arr[bytes - i - 1] = ch;
 }
}

For UINT64_C (0xabcdef1234567890), we will get the following sequence of bytes irrespective of platform.
ab cd ef 12 34 56 78 90

Infact, we can make this function more generic by adding a type parameter which will specify the sequence in which we need the bytes. Lets support little and big endian.

#define LITTLE 0
#define BIG 1

void
uint64ToByteArray (uint64_t num, size_t bytes, unsigned char *arr, int type)
{
size_t i;
unsigned char ch;
for (i = 0; i < bytes; i  )
 {
   ch = (num >> ((i & 7) << 3)) & 0xFF;
   if (type == LITTLE)
 arr[i] = ch;
   else if (type == BIG)
 arr[bytes - i - 1] = ch;
 }
}

So, if type == 0, byte sequence will be 90 78 56 34 12 ef cd ab
and if type == 1, byte sequence will be ab cd ef 12 34 56 78 90

2. Reconstruct uint64_t from bytes in little/Big endian sequence irrespective of the execution platform.

uint64_t
byteArrayToUInt64 (unsigned char *arr, size_t bytes, int type)
{
uint64_t num = UINT64_C (0);
uint64_t tmp;

size_t i;
for (i = 0; i < bytes; i  )
 {
   tmp = UINT64_C (0);
   if (type == LITTLE)
     tmp = arr[i];
   else if (type == BIG)
     tmp = arr[bytes - i - 1];

 num |= (tmp << ((i & 7) << 3));
 }
return num;
}

So, if byte sequence is 90 78 56 34 12 ef cd ab and type == 0, then uint64_t will be 0xabcdef1234567890

Little endian vs Big endian

Endianness in simple words is ordering of bytes in memory to represent some data. Computer memory in general is visualized as a sequence of bytes. But, in medium or high level languages like C we work with data types with size more than a byte.

Consider a 32 bit integer (in hex): 0xabcdef12
It consists of 4 bytes: ab, cd, ef, and 12. Hence this integer will occupy 4 bytes in memory. Say we store it at memory address starting 1000. There are 24 different orderings possible to store these 4 bytes in 4 locations (1000 - 1003). 2 among these 24 possibilities are very popular. These are called as little endian and big endian.

• Little endian - Stores the Least significant byte at the lowest address. Example: Intel Pentium Processors.
• Big endian - Stores the Most (Big) significant byte at the lowest address. Example: Sun/SPARC, IBM/RISC 6000.
  

On little endian system, memory will be like:

Address	Value
1000	12
1001	ef
1002	cd
1003	ab

On Big endian system, memory will be like:

Address	Value
1000	ab
1001	cd
1002	ef
1003	12

Now, the good news is that usually we don't need to care about endianness. It's taken care by Hardware Platforms, and Compilers. But, in some scenarios we need to care about endianness. A common scenario is when the data need to be exchanged between different systems. In such a situation, a standard layout is specified. Example: network protocols like TCP use Network Byte Order which is big endian. Thus the writers have to ensure that the data they write is in standardized order.

Before learning how to write/read in different orders lets play around and take a simple problem to determine endian of a given system.

There are 2 easy ways.

1. Write a known integer value in a binary file and view the file contents using a hex utility.

This method is more visual but is easy to implement and easy to detect layouts other than little/big endian. (Yes, there are other endians too like mixed endian).

Lets dump a 64 bit integer to a file:

#include <stdio.h>
#include <inttypes.h>

int
main ()
{
uint64_t integerForTesting = UINT64_C (0xabcdef1234567890);
FILE *fp = NULL;
fp = fopen ("dump.bin", "wb");
if (fp)
{
   printf ("Writing 64 bit number = 0x%" PRIx64 "  to dump.bin as it is\n", integerForTesting);
 fwrite (integerForTesting, sizeof (uint64_t), 1, fp);
 fclose (fp);
 printf ("Done writing\n");
 return 0;
}
printf ("Not written\n");
return 0;
}

Now, dump the contents of binary file (dump.bin) using command line hex utility like od.
#od -t x1 --width=8 dump.bin

If it dumps something like
0000000 90 78 56 34 12 ef cd ab
0000010
then the system is little endian.

If it dumps something like
0000000 ab cd ef 12 34 56 78 90
0000010
then the system is big endian.

2. Programmatically by looking at the value of byte stored at the starting address of a known number.

This method is limited in scope as it looks at only the first byte and only on the basis of that distinguishes between big and little endian. It won't detect other endian systems.

#include <stdio.h>
#include <inttypes.h>

int
main ()
{
uint64_t integerForTesting = UINT64_C (0xabcdef1234567890);
unsigned char *ch = (unsigned char *) &integerForTesting;
FILE *fp = NULL;
if (*ch == 0x90)
{
 printf ("this is little endian system\n");
}
else
{
 printf ("this is big endian system\n");
}
return 0;
}