Ubuntu GParted Question

[bostonman]

When I install a new drive in a Windows based PC (although I haven't for quite some time) it's usually just a simple format.

In Ubuntu I'm getting confused. Ubuntu doesn't recognize a blank (new) drive, so I figured out to use GParted. I've tried creating a new partition, but then also there is the option to create a partition table under device. Doing both doesn't seem to have an advantage, so I just create a partition (under partition drop down menu).

At this point Ubuntu doesn't see the new drive, so I have to unplug and reconnect it (sometimes more than once), and then it's finally recognized (a bit of annoying process).

After I use the command: dd if=file.img of=/dev/sdc status=progress

The end result (see attached image) looks as if the image is on one partition and a second partition exists. If I'm understanding correctly, this would mean I can't write any files to the partition that is 9.36GB in size because it's maxed, is this correct?

My assumption was that after I burn the image, I should have one 111.79GB long partition with only 9.36GB occupied.

[grumpydoc]

Is this a USB drive or an internal drive?

Your dd command will destroy whatever partition table you set up, but I'm not clear why you are trying to initialise the new disk from an image file.

You only have one partition on the disk which is 9.36G in size, the rest is unallocated.

[bostonman]

It's going to be a bootable drive (for an oscilloscope).

The image is a full Win98 OS, and the drive is an SSD connected via a USB convertor so I can burn the image.

I know what a partition is, but with Windows, I've never needed to create one within reason (I have to put Ubuntu as a secondary OS). With Windows, I basically format a X GB drive (let's call it 100GB), and I have 100GB worth of empty space.

From what I interpret, I have two partitions, which, in Windows, would be two separate drives. If so, then the 9.36GB partition can't have anything added to it.

[PKTKS]

When I install a new drive in a Windows based PC (although I haven't for quite some time) it's usually just a simple format.

In Ubuntu I'm getting confused. Ubuntu doesn't recognize a blank (new) drive, so I figured out to use GParted. I've tried creating a new partition, but then also there is the option to create a partition table under device. Doing both doesn't seem to have an advantage, so I just create a partition (under partition drop down menu).

At this point Ubuntu doesn't see the new drive, so I have to unplug and reconnect it (sometimes more than once), and then it's finally recognized (a bit of annoying process).

After I use the command: dd if=file.img of=/dev/sdc status=progress

The end result (see attached image) looks as if the image is on one partition and a second partition exists. If I'm understanding correctly, this would mean I can't write any files to the partition that is 9.36GB in size because it's maxed, is this correct?

My assumption was that after I burn the image, I should have one 111.79GB long partition with only 9.36GB occupied.

There is no possible comparison between MS WIN and *NIX.

The number of choices on *NIX are just orders of  magnitude above.

Gparted is handy for several things.. (like resizing etc)

But definitively  no *NIX folk  do blank disk installations using that..

The two tools for proper partition definition and alignment are
FDISK   for MBR  disk types
GDISK  for GPT  disk types.

ABOVE 1.5G  you are required to use GPT
otherwise MBRs are still a good choice for compatible reasons.

Just define a single partition using a console with one of these
tools and no problems should arise...

Format that with a proper FS ..
If the the OS itself did not perform a DISK SYNCH..

all tools come with a SYNCH option just after the write operation
which can be evoked any time by console..

Advice that this is dangerous stuff... 

** SAFETY WARNING **
DO NOT ISSUE A WRITE WITHOUT 100% SURE..

kinda shooting your own foot... it happens

Paul

[PKTKS]

It's going to be a bootable drive (for an oscilloscope).

The image is a full Win98 OS, and the drive is an SSD connected via a USB convertor so I can burn the image.

I know what a partition is, but with Windows, I've never needed to create one within reason (I have to put Ubuntu as a secondary OS). With Windows, I basically format a X GB drive (let's call it 100GB), and I have 100GB worth of empty space.

From what I interpret, I have two partitions, which, in Windows, would be two separate drives. If so, then the 9.36GB partition can't have anything added to it.

USE THE MBR FDISK based tool..
It is the proper tool and only that for such OS

Do not forget to place a proper MBR on the bottable disk
and flag that as "bottable"

Paul

[bostonman]

Okay, I'll try this tonight.

As for my assumption, this is two partitions with one being limited to just 9.36GB? Once installed in a Windows based system, it will show two drives, correct?

[grumpydoc]

Okay, I'll try this tonight.

As for my assumption, this is two partitions with one being limited to just 9.36GB? Once installed in a Windows based system, it will show two drives, correct?

Ignore PKTKS' advice for a sec - not that it's wrong in the more general sense but I think that it is wrong in this case, or not the whole story.

It looks like the image file is, as you say, the whole disk not just the C: partition, so it includes the partition table. It does not much matter that it was originally a smaller disk - you get the original partition and a whole load of empty space.

You *could* enlarge that partition to cover the whole disk (W98 has a 128GB limit per disk which you are just inside) but I advise against trying to do that on a bootable image with Linux tools. The W98 boot process is sensitive to the exact location/size of some disk structures - I remember digging out exactly which bits to edit, but it was always a bit hit and miss as I recall. Better to use one of the commercial partition editors or the Windows OS itself (though I don't recall if W98 is smart emough to re-size an in-use C: partition)

If you just want to format the unallocated space as the D: drive in windows that should be straightforward - select the unallocated space, add a FAT32/VFAT partition occupying the whole of the free space and format it with FAT32.

[Nominal Animal]

The
    /dev/sdc
block device represents the entire drive.  When you write to it (like you did with dd), you overwrite the contents of the drive, including any partition table and at least initial partitions (if any).

A block device can, but does not have to, contain partitions.  Even USB sticks can have partitions.  (If you use a DOS type partition table, aka MBR or Master Boot Record, and FAT32/VFAT file systems on each partition, all that happens is that when you stick the USB stick in, you'll see more than one logical drive appear.)

We do not know what your  file.img contains.  You can associate the image file with a loopback device using
    sudo losetup -fP file.img
where the -f flag tells losetup to use first free loopback device (it'll tell you what it is, say /dev/loop3), and -P flag to also scan its partition table.

After this, you can run gparted, pointing it to that particular loopback device, and modify the partitions, even add new ones.  To ensure you use the right device, run
    sudo gparted /dev/loop3
(or whatever loopback device losetup gave you).

After you have made changes (they are written back to the image file, so do keep a backup copy!), detach the loop device, and reattach (may give you a new loopback device!) using
    sudo losetup -d /dev/loop3
    sudo losetup -fP file.img
so that the new partition table is guaranteed to be in use.

You can format any new partitions you create; the new partitions will appear as /dev/loopNpP , where N is the loop device number (output by the losetup command), and P is the partition number.

When you're done, run
    sudo sync
    sudo losetup -d /dev/loopN

(The sync is just an extra precaution, and makes sure all modified files – including the image file – are actually written and not just cached in RAM.


In this case, you have already written the disk image to /dev/sdc.  (Nothing you did before that dd command matters.)
So, why not edit the disk partition table afterwards, using say gparted?  Just make sure you don't have any of its partitions mounted when you do.

What grumpydoc said is absolutely true: moving the existing partition may make the disk unbootable.  (See e.g. gparted manual for moving partitions and fixing system boot problems.)

[PKTKS]

please make it plain simple...

absolute trivial...


My old stuff is crammed in NBD virtual disks

nevertheless the procedure is _E_X_A_C_T _  the very same

fdisk /dev/sdx

make a single partition with bottable flag

Attached my NBD virtual disk  MBR

Paul

[grumpydoc]

please make it plain simple...

absolute trivial...


My old stuff is crammed in NBD virtual disks

nevertheless the procedure is _E_X_A_C_T _  the very same

fdisk /dev/sdx

make a single partition with bottable flag

Attached my NBD virtual disk  MBR

Paul

Yes, but from what bostonman has said, he has a disk image, containing a bootable copy of Windows 98 being the operating system and firmware for an oscilloscope - with which he needs/wants to initialise his new disk.

To do so just the "dd" command is sufficient (though I'd add a "sync" afterwards), he should be able to move the new disk straight to the 'scope and it will boot.

IF he wants to make the existing partition bigger, then the linux tools will do that but probably destry the ability to boot off the disk - so it is safer to use Windows tools for that.

[PKTKS]

... my first understanding was that he tried that before..

And the result was that misaligned partition...

Due to the evolution of disk geometries...  i HAD TONS ..
of such misaligned partitions when just trying to dd some
old image to new disks ...

Bottom line is that the dd image may boot .. or not
depending of factors like new disk tech (geometry) and
other factors..

My only one safe advice is:
-  DO THE PARTITIONING THE ABSOLUTE PROPER WAY:  fdisk  /dev/sdX
- save that and format the disk using FAT32 limits - bootable flag!!!
- *UNIX  does not care about bootable flag at all but MS will not boot wo that.
- mount your old image in whatever means possible..
- that depends on the applet used to image the file system.
- copy the file system over the new formatted partition..

then... the real safe part...  if that old stuff is 90s win9x... 
you need to:  C:> SYS C:   (or A:)     to actually have a bootable drive..

shitty as it always was  there no safer method..
but alternatives may be found... and enough to live with.

last   piece of desperate trying is converting that IMG to a virtual disk
and booting that in a suited VM so you can extract the file system
and also create some kind of bootable image to run SYS on that..

other methods  exist to put a boot loader on the MBR which will
found and boot  C:\io.sys    directly  ( SYSLINUX can do that )

The very first version of GRUB can also do that:

Code: [Select]

find --set-root --ignore-floppies --ignore-cd /io.sys
rootnoverify
chainloader +1
boot


Bingo grub boot that old thing magically wo SYS and DD
Paul

[SiliconWizard]

Next step is you need to resize the partition so it uses the unallocated space. GParted can do this.

[Monkeh]

... my first understanding was that he tried that before..

And the result was that misaligned partition...

Due to the evolution of disk geometries...  i HAD TONS ..
of such misaligned partitions when just trying to dd some
old image to new disks ...

What misaligned partition?

[grumpydoc]

... my first understanding was that he tried that before..

And the result was that misaligned partition...

The partition was not "misaligned" - it just didn't occupy the whole disk.

Which is what you'd expect when copying the whole disk image from a smaller disk to a larger one.

Due to the evolution of disk geometries...  i HAD TONS ..
of such misaligned partitions when just trying to dd some
old image to new disks ...

Yes, back in the mists of time when disks were read using C/H/S (Int 13 function 2/3) addresses you had to be much more careful - but logical block addressing has been the norm for, what?, 25 years now at least.

Bottom line is that the dd image may boot .. or not
depending of factors like new disk tech (geometry) and
other factors..

My only one safe advice is:
-  DO THE PARTITIONING THE ABSOLUTE PROPER WAY:  fdisk  /dev/sdX
- save that and format the disk using FAT32 limits - bootable flag!!!
- *UNIX  does not care about bootable flag at all but MS will not boot wo that.
- mount your old image in whatever means possible..
- that depends on the applet used to image the file system.
- copy the file system over the new formatted partition..

then... the real safe part...  if that old stuff is 90s win9x... 
you need to:  C:> SYS C:   (or A:)     to actually have a bootable drive..

OK, yes - you are right about the SYS command.

But you need to have W98 booted to use SYS to copy the boot files over to a new drive - which presents you with a catch-22 situation....

shitty as it always was  there no safer method..
but alternatives may be found... and enough to live with.

Paul


last   piece of desperate trying is converting that IMG to a virtual disk
and booting that in a suited VM so you can extract the file system
and also create some kind of bootable image to run SYS on that..

other methods  exist to put a boot loader on the MBR which will
found and boot  C:\io.sys    directly  ( SYSLINUX can do that )

You can create a bootable DOS disk that way, but Windows hasn't booted via DOS since 3.11 days, just copying the files to a freshly formatted disk is not sufficient - there is some special sauce (I did know how to do it waaaay back but would have to dig around and see if Google can refresh my memory and life is almost certainly too short).

However I would expect a disk image which has been created with dd to boot just fine, however the partition will not automatically fill a new, larger, disk.

Bostonman was incorrect in his original assumption that transferring the old image with dd would leave him with a large ~110Gb partition with 9G or so used. What he got is a 9Gb partition which was full and ~100Gb of unallocated space on the disk - but that's what you would expect.

I'd suggest the following steps, assuming that the target 'scope is available.

• dd the image as described, follow that with a sync command to flush the blocks to disk and transfer to the 'scope
• if it boots, great
• if not you've got other problems like C/H/S *does* matter to the 'scope or a corrupt image and you *will* need to figure out how to manually copy the files and enable the Windows boot process, or you have a corrupt image
• Shut the scope down and put it back in the Ubuntu machine, use Gparted to enlarge the 9Gb partition to fill the disk
• Put it back in the 'scope, if it boots then you're home and dry
• If not then Gparted moved some critical files. IIRC the real issue was moving a partition but Google should be able to cough up the details of how to patch the image.
  

Next step is you need to resize the partition so it uses the unallocated space. GParted can do this.

Yes, it can though if any boot files need to be in a fixed spot on the disk and they get moved it might not boot - that said, thinking about it for a fraction longer, ISTR that the big issue was moving the start of a partition - there's an offset stashed somewhere iin the early boot loader which needs updating.

[PKTKS]

Next step is you need to resize the partition so it uses the unallocated space. GParted can do this.

not always...

sometimes (very frequently) gparted will just not be able to
handle that "phantom" space or misalignment that you will
only see there as well...

doing the partition table on fdisk will never show you those phantoms.

seems a gparted exclusive "feature" ...

But Mileage may vary...

Paul

[SiliconWizard]

I've never seen that happen. Unless maybe the copied partition is a mess to begin with.

[PKTKS]

I've never seen that happen. Unless maybe the copied partition is a mess to begin with.

Not necessarily...

As said above old geometry  having different "boundaries"
when just "dd" over a device with different C/H/S  **MAY**  hit
misaligned boundaries on the partition end (if aligned by
Meg or by cylinder)  or.. show these small phantoms between
partitions

there is no possible way to recover that discrepancies in usable
space  unless doing a new partition schema

It is not about "free space" ... the partition boundaries do not align
exactly between devices..

It is harmless but very annoying when presented..

Paul

[bostonman]

Sorry I've been quiet, I got sidetracked with an issue and haven't had time to try any suggestions.

I'll need to read through this and try the suggestions tomorrow.

As for my intentions, I didn't realize this was such a loaded question. The file I have was compressed in Linux, so it's one reason I've been working in Linux. I thought the process would be as simple as burning the image onto any hard drive (providing it has the minimum size to fit the image), and the image would over write the entire drive (within reason) leaving me with a bootable drive and remaining empty space (in this case, rounding up, and using ideal numbers, 120GB - 10GB, or 110GB).

Past experiences with using Gparted was that I'd burn the image, and then the drive wouldn't boot because the image got burned onto another partition (or maybe it wasn't flagged as bootable). After basically getting lucky, I'd burn the same image and the drive would successfully boot.

[Nominal Animal]

I thought the process would be as simple as burning the image onto any hard drive (providing it has the minimum size to fit the image), and the image would over write the entire drive (within reason)

No, unfortunately: the image also specifies the size of the file system on the drive.

You should be able to just resize the file system afterwards using many tools, gparted included.  This should not cause the file system to become unbootable.  (It can occur if the partition is not aligned to a 4k boundary, and the tools move it.)  You can also use Windows tools to resize the partition, although I don't use Windows so I cannot suggest any particular tool.

If the image contains an extended partition (it will be marked as such – it is a container for secondary partitions, MBR partition table having room for only four primary partitions), then you do need to resize the extended partition first to the full drive size, before you can extend the secondary partitions inside it.

[bostonman]

You should be able to just resize the file system afterwards using many tools, gparted included.  This should not cause the file system to become unbootable.  (It can occur if the partition is not aligned to a 4k boundary, and the tools move it.)  You can also use Windows tools to resize the partition, although I don't use Windows so I cannot suggest any particular tool.

The problem I initially had: the hard drive was fresh out of the box. Simply connecting it to Ubuntu (SATA to USB converter) didn't show the drive. I'll admit my computer experience never dealt with a "blank" drive. If anything, I'd boot a PC with a Windows CD and install Windows that way. Maybe back in the DOS days I dealt with a fresh drive, however, I don't remember, and things have changed.

In any case, I searched online and learned about Gparted. I thought once I format the drive, that would allow (in this case) DD to copy the image thus making it a normal bootable drive. I don't remember the steps, but at some point the drop down menu wouldn't be highlighted to format, other times I'd have to create a partition before formatting, etc... Afterwards, when I finally had what I thought was a formatted drive, I couldn't see it in Terminal.

I'd unplug the USB and reconnect it, and, for whatever reason, eventually the drive would appear with some odd name whereas in Windows I could name the drive.

Anyway, those were the steps I took. Once I saw a 10GB partition and a separate 110GB partition, I "assumed" that it's A: two separate drives since it's two partitions, and B: I couldn't write anything to the 10GB partition because the extracted image occupies exactly the size it needed.

Reading through all the advice, it seems the anomaly is that I'm trying to burn a bootable image and have the remaining space available without taking extra steps.

I guess the steps I should take (from what I interpreted) is to burn the image as usual, attempt to resize the partition thus making the other partition smaller, and still ending up with two partitions, only the first (bootable) one will be resized.

[Monkeh]

Okay, so there's a lot of basic misunderstanding going on here.

The problem I initially had: the hard drive was fresh out of the box. Simply connecting it to Ubuntu (SATA to USB converter) didn't show the drive.

This is because it's an unpartitioned drive with nothing on it - there's nothing for unprivileged tools to do with it. The kernel log ('dmesg') will show you a nice long string of messages culminating in the detection of a new block device.

In any case, I searched online and learned about Gparted. I thought once I format the drive, that would allow (in this case) DD to copy the image thus making it a normal bootable drive.

Completely unnecessary because your image includes the partition table.

Anyway, those were the steps I took. Once I saw a 10GB partition and a separate 110GB partition, I "assumed" that it's A: two separate drives since it's two partitions, and B: I couldn't write anything to the 10GB partition because the extracted image occupies exactly the size it needed.

You only have one partition. The rest of the space is unallocated. What you have is an exact copy of the original drive contents plopped into a larger space.

All you need to do is write the image to the raw device (/dev/sdx, where x is the correct device and not the wrong device), pull the drive, and put it where it belongs. Everything should work fine. If you need to enlarge the partition instead of simply adding a second one (which you can do from Windows..), well, I leave advice on that to people familiar with working on 20 year old half-functional OSes.

Most importantly in all of this: Don't listen to PKTKS. He's not paying attention and will insist on leading you on wild goose chases to satisfy his strange urges.

[Nominal Animal]

There is nothing wrong in Monkeh's suggestion, but here's mine:

• Run
      lsblk
  or
      bash -c '(cd /sys/block && for D in */device/model ; do echo -n "${D%%/*}: " ; cat "$D" ; done)'
  to determine which device is the correct target device.  For SATA and USB drives, it will be /dev/sdX, for NVMe drives it will be /dev/nvmeXnY.  For simplicity, I will use /dev/sdX for the device name.  Partitions will be named (numbered) /dev/sdXP and /dev/nvmeXnYpP, respectively.
   
• Write the image over the drive using e.g.
      sudo dd if=image.img of=/dev/sdX bs=2M status=progress
  Remember that this will destroy any partition tables and possibly any data on the drive.  The bs=2M is just an optimization, telling dd to use two megabyte chunks, as larger block sizes like this tend to yield optimum I/O speeds.  The final block may be smaller; the image does not have to be a multiple of two megabytes in size.
   
• Optionally, make sure the data is written to the device, and not just cached in RAM, by running
      sudo sync
  and make sure none of the partitions on it are mounted, by running
      sudo umount /dev/sdX*
   
• Usually, the drive now has only one partition, which does not cover the entire drive.
  Use your favourite partition manager program to resize it, so it covers the entire drive.  For example,
      sudo gparted /dev/sdX
  
  Do not try to move the partition (change its start), only resize (change its end or size) the existing one to cover the entire disk.  If asked, decline moving the partition.  Moving the partition is what usually causes the drive to become unbootable in MS-DOS or Windows; resizing should not affect bootability.
  
  Unused area or unpartitioned area shown in the partition manager program refers to the part of the disk that is not covered by any partitions.  It cannot be mounted, and it does not show up as a separate drive in DOS/Windows.  It is just unallocated, unused space on the drive.
  
  If you have multiple partitions, you may have an Extended partition.  This is a container in MBR partitioning scheme, and must be resized to fill the rest of the drive before any partitions within it can be extended to cover the entire drive.  Some partition manager programs do this automatically for you, others limit the size of partitions within it to the size of the extended partition.
   
• Run
      sudo sync
  to ensure changes are written to the device, and
      sudo partx --add /dev/sdX
  to ensure the kernel knows about the partitions on that device.  The latter may cause the partition(s) to be automounted, so before detaching the device, unmount the partitions either via your desktop (ejecting the drive), or by running
      sudo umount /dev/sdX*
   
• You can now either mount the device (if not automounted in the previous step), or detach it (but do check your desktop if it automounted it; do not detach mounted drives, as that can cause the data on the drive to become garbled) and attach to the target machine.
   

[PKTKS]

My advice is simple:
-  IF IT  JUST **WORKS**  than go with it..

You can try just DD and see.. if it boots keep it..
if not follow the advices...

#Nominal  placed some very good tips and explanations
about the file system boundary results on the partition...

If it just works go with it.. if not follow the tips..

Alas there is no hidden agenda whatsoever as patronized (!$#_)

Paul

[bostonman]

Run
    sudo sync
to ensure changes are written to the device, and
    sudo partx --add /dev/sdX
to ensure the kernel knows about the partitions on that device.  The latter may cause the partition(s) to be automounted, so before detaching the device, unmount the partitions either via your desktop (ejecting the drive), or by running
    sudo umount /dev/sdX*

I followed your steps but forgot to do the above. In any case, the drive still had two partitions, or maybe one wasn't a partition, but simply unallocated. I resized the bootable partition (the end of the partition, the ride side, not the beginning), and it made a single line (i.e. one) partition after I maxed out the right side.

After I connected it in the scope (via a IDE to SATA converter that I just bought) and the scope booted. I took a picture of the disk space for reference (see attached). Sadly this is a waste of a drive because I'm not writing 120GB of data to it, however, it wasn't expensive. Any smaller sized drives which seemed hard to find weren't much cheaper.

[PKTKS]

back to the points...

There are 2 things (distinct ones):
- the partition size
- the file system size.

They MAY  not be  measured with same metrics...
While one is based on C/H/S and LBA converted tables
the other is measured is bits/bytes and Meg/Gig (aka base 2)...

When properly done they align..
when done like this case..  using DD over a new device...

there may be problems in alignment...

You first extend the partition boundary to the right... OK
this seems already done..

Then you need to extend the file system size to the (NEW) partition boundary

Here is the point of problems..
The tool you use is different and it will need to rebuild the
whole allocation table of your files...

There are chances of serious problems.. or may be not.
Using a well written software it should be OK

and *MAY BE*  just a small phantom arises at the end of the boundary.

May be not..

Paul


 

[Nominal Animal]

Run
    sudo sync
to ensure changes are written to the device, and
    sudo partx --add /dev/sdX
to ensure the kernel knows about the partitions on that device.  The latter may cause the partition(s) to be automounted, so before detaching the device, unmount the partitions either via your desktop (ejecting the drive), or by running
    sudo umount /dev/sdX*

I followed your steps but forgot to do the above.

No harm done, I believe: Linux partition managers I know all do an fsync on the device (a more targeted sync), so the sync command is optional.  And if you detached the drive, there was no reason for the Linux kernel to know about the partitions anyway.  (The reason I included partx in the steps is that if one uses it, one does not need to detach and reattach the drive.)  And because the modified partition didn't get automounted, there was no need to unmount anything.

Happy to see the drive booted in Windows, and is now working; thanks for the screenshot!

[bostonman]

Thanks for all the help!

I’m guessing since it booted that I won’t get any odd behavior at some point later, so I’m assuming this has been resolved.

[PKTKS]

    sudo sync
to ensure changes are written to the device, and
    sudo partx --add /dev/sdX

this is absolute right
each one syncs a target..