Multi-Boot Application - Progress

Contribute code to SolydXK and make it even better.
User avatar
ScottQuier
Posts: 1781
Joined: 18 Jul 2013 15:55
Location: Newport News, VA

Multi-Boot Application - Progress

Postby ScottQuier » 27 Jul 2014 12:21

A picture of the application as it stands right now:
TheRestOfTheStory.jpg
And, this is the little dialog I created for a bit of embedded help, figuring that the purpose of all the controls may not be intuitively obvious.
HelpDialog_01.jpg
HelpDialog_01.jpg (47.42 KiB) Viewed 6794 times
Here's what I've got working so far:
  • On startup:
    • It gets the list of currently available for download iso and parses it to a python list for use in a number of places:
    • The ISOs for Download window
    • The Down Load ComboBox
  • The "Select USB" ComboBox is populated with the list of mounted USB mass-storage devices
  • The "Refresh" button works. If you start the application before mounting your USB device, click this button after mounting and the ComboBox re-populates. Same if you remove a device; the refresh button will remove that device from the ComboBox.
  • As you can see, the "Show All ISOs" button works - this is old news
  • The RadioButton group works as one would think - defines which operation the "Execute" button will perform.
  • The USB Free Space data entry show how much space is available on the selected USB device. This, of course, gets updated as one selects a different item from the top ComboBox.
  • The FileSelection button opens a file selection dialog, with a filter in place to allow one to see only *.iso files and thus only select the same. When the user selects such a file, the associated TextEntry control is populated with the full path, just for the user's benefit. Re-Worked - I couldn't get the FileSelectorButton to work reliably (especially the 'Cancel' button in the generated dialog) so I re-worked this to, instead, be a standard button that, in turn, invokes a FileChooserDialog (which is defined in the glade file). From that dialog, I can return the user's response (Open or Cancel) and respond in a correct manner.
  • If the size of the selected file (whether from the "Add ISO" or from the "Down Load") is larger than available space, the "Execute" button is turne d off.
  • Implemented a status bar - with correctly behaving Push/Pop features.
Yet to be done
  • In all my goofing around ... umm ... testing, I've discovered that putting a GPT partition table on a device and then, later, changing to MSDOS partition table causing issues when installing grub. It's something about multiple partition labels not being supported by the current version of grub (or, maybe just grub-install). So far, the only thing I've found that corrects the situation is wiping out the entire device with

    Code: Select all

    dd if=/dev/zero of=/dev/sdX bs=512
  • [DONE]Check for network connection. Turn off all the download related options if network connection is not found. Maybe let the user know why they are disabled?
  • [DONE]Actually generate a multiboot USB so I can .... - I used Schoelje's script ( http://forums.solydxk.nl/viewtopic.php? ... 675#p34009 ) to build the USB. Now I can use it to test...
  • Build the multiboot USB from inside the code!
  • [DONE] (30JUL2014) Download and save the selected ISO to the selected USB device.
  • [DONE]Build and test the functionality of the "ISOs on USB". How Done - Remove the button on the "Delete". Instead, when the user selects the "Delete ISO" RadioButton, the application inspects the selected USB device looking for *.iso files in the root of the device and returns that list as a list of valids displayed in the ComboBox.
  • [DONE] (30JUL2014) Need to add an "Empty" selection to the "Delete" ComboBox - just to make the user more comfortable.
  • [DONE] (30JUL2014) Delete ISO (from the USB device, don't you know).
  • [DONE] (01AUG2014) Need to add some way to get user confirmation prior to completing the delete operation.
  • [DONE] (01AUG2014) Completely clean the USB device
  • [DONE]Put another TextEntry box next to the "USB Free Space" one to display the amount of space required for the selected ISO. My son, very wisely, pointed out that users will want to know not only how much is available, but also how much is required ... expecially when they select one and the "Execute" button goes ghosted. This will also require changing the label to left for that line or just re-working that whole section a bit.
  • Reconsidered Not going to implement this - it doesn't make much sense. I'm also thinking about adding a menu to this if I can determine that it makes sense. At the very least, I need to provide some sort of immediate "Help" or instructions so the user doesn't have to guess. In particular, this thing will only write one ISO to the USB device for each click of the Execute button. You want two on the USB, you have to go through the process of selecting the source and clicking the button again.
  • [Done] There's a few formatting (control location) issues that I want to address.
  • Need to make the grub menu a bit cleaner. It currently lists all possibilities. Need to make it include only those ISOs that are actually on the USB device. Modify it when the user add/deletes and ISO and re-install grub to make it work.
  • I've done nothing towards implementing localiazation. It's all in English and that won't fly in the file product.
Something I discovered yesterday. The first line of my python scripts is

Code: Select all

#! /usr/bin/env python3
and I thought this would cause the program to execute in python3. Well, if one runs

Code: Select all

$python program.py
on a SolydXK system, this actually will run in python 2.7.

However, if one changes the script to be executable and then

Code: Select all

$ program.py
it runs in python3.

This is because:

Code: Select all

$ cd /usr/bin
$ ls -al python
lrwxrwxrwx 1 root root 9 Jul 19 17:42 python -> python2.7
$ ls -al python3
lrwxrwxrwx 1 root root 9 Jul 19 17:42 python3 -> python3.4
My mis-understanding this caused me to have to re-work code that was working in 2.7 so that it would run in python3.
Last edited by ScottQuier on 02 Aug 2014 14:33, edited 1 time in total.
Reason: Update of status and the pictures
Scott
Quoting zerozero, "The usage of PPA's in debian-based
systems is risky at best and entails serious compatibility
problems; usually it's the best way to destroy an install"

Deleted User 2764

Re: Multi-Boot Application - Progress

Postby Deleted User 2764 » 27 Jul 2014 13:43

You sure have beeen busy. Two things came to mind:

1. If someone is trying to do this without an internet connection, will the code throw a fatal error if it can't download the list of ISOs on startup? Any error catching if someone clicks to view the list of available downloads and didn't notice their internet connection had died?

2. Please forgive if you already know this. I thought I'd mention it here, maybe it might help others too. I haven't tried Glade 3.18/Python3 yet but from my tutorial here, it seems I got it to work by doing this:

Code: Select all

#!/usr/bin/env python3

from gi.repository import Gtk
Then to specify the glade file I did this:

Code: Select all

    def __init__(self):
        self.gladefile = "my_glade_file.glade"
        self.builder = Gtk.Builder()  # Note we use Gtk.Builder() and not gtk.Builder()
        self.builder.add_from_file(self.gladefile)
Also Schoelje told me this is the way to set it up if you're going to be translating the application in different languages:

Code: Select all

#!/usr/bin/env python3
#-*- coding: utf-8 -*-
Of course in that case, you'll need to also install poedit. In addition, he mentioned using python-stdeb if you are going to make it into a debian package. But I don't know how to use that (yet).

Does it still work that way in Glade 3.18?

User avatar
ScottQuier
Posts: 1781
Joined: 18 Jul 2013 15:55
Location: Newport News, VA

Re: Multi-Boot Application - Progress

Postby ScottQuier » 27 Jul 2014 17:08

1. Your #1 is a very good catch and I've added it to the "ToDo" list.

2. My first lines in the python scripts are the same as you've outlined, except I had a space between the ! and /. I corrected that, but it didn't make any difference. No worries.

Glade 3.18 vs 3.16 - no impact on this project. I started the glade file with 3.16 and I've been working in 3.18 since the Jul UP.

BTW 3.18 is soo much better than 3.16!

Your code for initializing the GUI is (nearly) identical to what I'm using .... so that's a good thing.
Scott
Quoting zerozero, "The usage of PPA's in debian-based
systems is risky at best and entails serious compatibility
problems; usually it's the best way to destroy an install"

User avatar
ScottQuier
Posts: 1781
Joined: 18 Jul 2013 15:55
Location: Newport News, VA

Re: Multi-Boot Application - Progress

Postby ScottQuier » 27 Jul 2014 17:23

@Schoelje ....

Please refer to your thread --> http://forums.solydxk.nl/viewtopic.php?f=33&t=3675
  1. I was able to build a strictly msdos partition tabled USB and it booted fine in both legacy and EFI.
  2. I was also able to build a gpt partition tabled USB device and got it to boot in legacy without any issue, but not EFI.
    Command used were:

    Code: Select all

    # parted -a optimal -s /dev/sdf mklabel gpt
    # parted -a optimal /dev/sdf mkpart non-fs 0% 1% set 1 bios_grub on
    # parted -a optimal /dev/sdf mkpart fat32 1% 100% 
    # mkdosfs -F32 -n 'SOLYDXK' -I /dev/sdf2
    
    Note that I formatted the second partition and didn't worry about the first.
    This genereted the following structure

    Code: Select all

    Model: Generic STORAGE DEVICE (scsi)
    Disk /dev/sdf: 4025MB
    Sector size (logical/physical): 512B/512B
    Partition Table: gpt
    
    Number  Start   End     Size    File system  Name   Flags
     1      1049kB  2097kB  1049kB  ext4         ext3   bios_grub
     2      2097kB  4024MB  4022MB  fat32        fat32
    
    For the EFI, it stopped at the grub> prompt with an error message about a missing file (if I remember correctly)
  3. I would love to see what you've attempted and what paths you ventured down. That information could inspire a thought or two on my part.
  4. You indicated that Mac users will need the device to be GPT. Is there any requirement for the filesystem used? Will they need EFI boot capability?
  5. You said that the EFI files are Ubuntu generated. Do you have the source? Do you know how they are supposed to be built; what commands are to be used or is it a "simple" compile using their makefile, etc?
  6. I'm going to include some code in the application that (re-)generates the /boot/grub/grub.cfg to include menu listings for only those ISOs on the USB device. Of course, after it's been changed, I'll have to run the grub-install command (again). As it stands now, the grub menu is just too busy and a user could get very confused - especially if he/she selects an item on the menu for which there isn't an ISO - that leads to a grub error and that's just messy.
Scott
Quoting zerozero, "The usage of PPA's in debian-based
systems is risky at best and entails serious compatibility
problems; usually it's the best way to destroy an install"

Deleted User 2764

Re: Multi-Boot Application - Progress

Postby Deleted User 2764 » 27 Jul 2014 19:36

@Scott: Thanks for the extra info! I'm glad that 3.18 is much better. Now maybe when I get time I can finally get to that Python/Glade tutorial! It probably won't be for awhile yet as I first want to get through the EdX Introduction to Linux course that starts on Friday. Maybe after I finish that course.

Great news that the error checking is on the To Do list. I assume you'll want people to test it as well eventually. I may still be able to do some small testing here and there so I might be tempted to try it out.

When developing a program, I try to think of all the ways the program can break, then break it and fix the bugs. But as we all know, we can't always think of everything! :)

User avatar
Arjen Balfoort
Site Admin
Posts: 9223
Joined: 26 Jan 2013 19:36
Location: Netherlands
Contact:

Re: Multi-Boot Application - Progress

Postby Arjen Balfoort » 28 Jul 2014 14:40

@ScottQuier
2) Just guessing, but did you set the "boot" flag?

3) I never pursued gpt. Perhaps it's better to create a separate topic for this.

4) Sorry, I don't have a Mac, and don't know anyone who has one, and knows what is EFI... Perhaps leave the GPT feature as it's a "nice to have".

5) I took them from the LMDE ISOs. So, even Clem took them from Ubuntu.

This is what I explored for building the Grub EFI binaries: http://wiki.osdev.org/GRUB#Configuring
but I never was able to boot an ISO or USB with my own compiled files.

I saved my commands:

Code: Select all

grub-mkimage -d . -o bootx64.efi -O x86_64-efi -p /boot/grub `find *.mod | xargs | sed -e 's/\.mod//g'`
# make it recognizable as EFI (only on EFI machine)
efibootmgr -c -d /dev/sdc -p 2 -l \\EFI\\BOOT\\bootx64.efi -L "SolydXK" -v


SolydXK needs you!
Development | Testing | Translations

User avatar
ScottQuier
Posts: 1781
Joined: 18 Jul 2013 15:55
Location: Newport News, VA

Re: Multi-Boot Application - Progress

Postby ScottQuier » 30 Jul 2014 00:53

@Schoelje

2 - See the "parted -l" output above. You can only do one flag on a partition and GPT required the bios_grub flag set. Doesn't really matter now as the whole GPT on EFI thing is for a later release.

5 - I gave that link a look. I'm thinking that's something I'll work on this weekend. Who knows, I might get it to compile and boot ... I could get lucky. :)

Anyway, here's progress (I've updated the OP as well)
  • Re-worked the layout of some of the controls. Not perfect, but I think it's a bit better.
  • Re-worked the whole idea of the "Delete ISO" functionality. When the user selects this option, the associated ComboBox is populated with the names and sizes of the ISOs on the selected USB device.
  • In addition to showing how much room is left on the USB device, I'm also showing how much space is required to copy the selected ISO (either from the DownLoad or from the user's machine) to the selected USB device.
  • I was going to ghost the "Execute" button if not all required data was available or if the needed space > the available space. Grew too cumbersome. Instead, let the user do what he/she wants. When they click the button, the app looks at the selected options and associated data and throws up a message dialog if there is one or more things wrong with the select.
Here's a couple of screen shots.
Multi_DeleteDropDown.jpg
Showing the content of the Delete ComboBox
MultiPopulated.jpg
And this is how it might look part way through the process, especially if the user isn't quite sure what he's up to.
@RavenLX - I've been running the MATE DE. Glade runs real sweet on MATE. At one point today, it was necessary for me to log into Xfce to do some quick testing. Anyway, spun up Glade to make a couple of quick changes to the GUI and glade crashed on me. In MATE, it's not crashed yet. Hmmmm....
Scott
Quoting zerozero, "The usage of PPA's in debian-based
systems is risky at best and entails serious compatibility
problems; usually it's the best way to destroy an install"

Deleted User 2764

Re: Multi-Boot Application - Progress

Postby Deleted User 2764 » 30 Jul 2014 04:12

I run mostly in KDE. We'll see how that goes. :)

User avatar
ScottQuier
Posts: 1781
Joined: 18 Jul 2013 15:55
Location: Newport News, VA

Re: Multi-Boot Application - Progress

Postby ScottQuier » 03 Aug 2014 18:35

OK then .... testing did not go so well. I started testing and discovered one problem after another. The interface is pretty good (there are a couple of implementation issues I have to iron out) but the code has me really worried.

I have to figure out:
  • How to get feedback to the user while in the middle of series of time consuming steps
  • How to make sure the USB device has a good and usable partition table. This is necessary because the user may be coming back to the table with a device that's already got one iso installed on it and he/she want to install another. Or the user may be running the application with garbaged device and I have to recreate the partitioning scheme from the ground up. Yep - fun stuff.
  • How to implement a dictionary in python. I tried to use lists the first time, but that got to be very cumbersome. Dictionaries have a good chance of being easier to use.
Yep - it could be another couple of weeks before I'm back to the testing phase. Gotta love it!

@Schoelje - yes, I'm still having fun!

Time for a snack and then a good workout!
Scott
Quoting zerozero, "The usage of PPA's in debian-based
systems is risky at best and entails serious compatibility
problems; usually it's the best way to destroy an install"

User avatar
Arjen Balfoort
Site Admin
Posts: 9223
Joined: 26 Jan 2013 19:36
Location: Netherlands
Contact:

Re: Multi-Boot Application - Progress

Postby Arjen Balfoort » 03 Aug 2014 18:56

How to get feedback to the user while in the middle of series of time consuming steps
You need to run time consuming processes in a separate (classed) thread.
Any messages you want to return to the UI, you need to pass in a queue.
If you want I can help you with that.
How to make sure the USB device has a good and usable partition table.
I'm not sure why you want to do that, but can you let the user choose to recreate the partition table, and warn him that he'll loose all data in the process?
How to implement a dictionary in python.
Doing from memory:

Code: Select all

mydict = {}
mydict['name'] = "Arjen"
mydict['nickname'] = "Schoelje"
print(("%(name)s nickname is %(nickname)s" % { "name": mydict["name"], "nickname": mydict["nickname"] }))
Iterating through a dictionary is a bit different then iterating through a list.


SolydXK needs you!
Development | Testing | Translations

Deleted User 2764

Re: Multi-Boot Application - Progress

Postby Deleted User 2764 » 03 Aug 2014 19:10

Autodetection of partitions is a good idea if you're adding another ISO.

However, how about you do a directory search for *.iso and if it shows it you know you have a partition?

And as Schoelje said, if not, then to ask if you want to repartition/reformat or use the drive as-is?

If the USB stick is bad, I don't know how to detect that. I do remember that sometimes RedoBackup, for example, would end up a write error (after already starting the process) near the end of the process or in the middle. The problem was the SD Card (in this case), while it had data on it, did not have a partition. It needed an NTFS partition so I had to backup the data, repartition, relabel and replace the data in order to use it for RedoBackup as well.

In conversation with a friend of mine, I learned that USB and SD cards are not like hard drives. They are actually memory. So while you can "partition" it, you're not actually partitioning the USB stick or SD card. It is like a virtual paritioning instead. I don't know if that bit of technical information may help, but maybe it might point in a direction on how to detect virtual partitions on a memory type device such as USB stick or SD card, which is different from detection of partitions on actual hard drives?

User avatar
ScottQuier
Posts: 1781
Joined: 18 Jul 2013 15:55
Location: Newport News, VA

Re: Multi-Boot Application - Progress

Postby ScottQuier » 03 Aug 2014 19:12

Schoelje wrote:
How to get feedback to the user while in the middle of series of time consuming steps
You need to run time consuming processes in a separate (classed) thread.
Any messages you want to return to the UI, you need to pass in a queue.
If you want I can help you with that.
Example code or a link to some example code would be good.
Schoelje wrote:
How to make sure the USB device has a good and usable partition table.
I'm not sure why you want to do that, but can you let the user choose to recreate the partition table, and warn him that he'll loose all data in the process?
In my testing and work along the way, I've found a number of ways in which to garbage up the partition table and/or the partition and still have it recongized by Linux but not be suitable for booting or for installing grub.

I think I know how to get it done, I just have to be a bit more careful in my design than my first shot at it. Somehow, I was able to totally trash the display of my DE (thankfully not my install). A quick reboot and all was good again. But, I have to figure that one out so that I can make sure it doesn't happen.
Schoelje wrote:
How to implement a dictionary in python.
Doing from memory:

Code: Select all

mydict = {}
mydict['name'] = "Arjen"
mydict['nickname'] = "Schoelje"
print(("%(name)s nickname is %(nickname)s" % { "name": mydict["name"], "nickname": mydict["nickname"] }))
Iterating through a dictionary is a bit different then iterating through a list.
Yep - on all counts. I just have to give it a play and see if it works better than lists for my purposes. More on this later as I know more.
Scott
Quoting zerozero, "The usage of PPA's in debian-based
systems is risky at best and entails serious compatibility
problems; usually it's the best way to destroy an install"

User avatar
Arjen Balfoort
Site Admin
Posts: 9223
Joined: 26 Jan 2013 19:36
Location: Netherlands
Contact:

Re: Multi-Boot Application - Progress

Postby Arjen Balfoort » 03 Aug 2014 19:45

Take a look at: https://github.com/SolydXK/updatemanage ... /mirror.py

This is a threaded class which is called from: https://github.com/SolydXK/updatemanage ... gerpref.py

The class's structure:

Code: Select all

#! /usr/bin/env python3
#-*- coding: utf-8 -*-

import threading


class MirrorGetSpeed(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)

    def run(self):
        # Do your thang
In updatemanagerpref.py:

Code: Select all

t = MirrorGetSpeed(self.mirrors, self.queue, self.umglobal)
self.threads[name] = t
t.daemon = True
t.start()
self.queue.join()
GLib.timeout_add(5, self.checkThread, name)
The queue is handled in the function checkThread.


SolydXK needs you!
Development | Testing | Translations

User avatar
ScottQuier
Posts: 1781
Joined: 18 Jul 2013 15:55
Location: Newport News, VA

Re: Multi-Boot Application - Progress

Postby ScottQuier » 03 Aug 2014 21:38

I've cloned the entire project - now I have a clue where to look. Thanks!
Scott
Quoting zerozero, "The usage of PPA's in debian-based
systems is risky at best and entails serious compatibility
problems; usually it's the best way to destroy an install"

User avatar
Arjen Balfoort
Site Admin
Posts: 9223
Joined: 26 Jan 2013 19:36
Location: Netherlands
Contact:

Re: Multi-Boot Application - Progress

Postby Arjen Balfoort » 03 Aug 2014 21:45

If you have trouble with threads in Python (I still have), just push your project to Github, and I'll take a look.


SolydXK needs you!
Development | Testing | Translations

User avatar
ScottQuier
Posts: 1781
Joined: 18 Jul 2013 15:55
Location: Newport News, VA

Re: Multi-Boot Application - Progress

Postby ScottQuier » 03 Aug 2014 21:58

Schoelje wrote:If you have trouble with threads in Python (I still have), just push your project to Github, and I'll take a look.
You may have a bit of a wait for this to happen .... I'm kinda stubborn :)

But, I'm not stupid either - I'll chew on it for a couple or three day (I have a lot things I want to learn a bit better) and then I'll ask for help.

Thanks!
Scott
Quoting zerozero, "The usage of PPA's in debian-based
systems is risky at best and entails serious compatibility
problems; usually it's the best way to destroy an install"

Deleted User 2764

Re: Multi-Boot Application - Progress

Postby Deleted User 2764 » 04 Aug 2014 14:15

@Scott: Maybe you could also take a look at this?

http://www.tutorialspoint.com/python/py ... eading.htm

It's a tutorial on python multithreading. It looks more like a reference than a tutorial though, but maybe you'd find it useful?

User avatar
ScottQuier
Posts: 1781
Joined: 18 Jul 2013 15:55
Location: Newport News, VA

Re: Multi-Boot Application - Progress

Postby ScottQuier » 07 Aug 2014 23:06

Well, I've spent the last couple or three day (OK, 5 days) studying python a bit. Figured out how it does classes and inheritance. So, for some of the stuff that was causing the code in Ver 0.0.01 to look really, really ugly, I've created a base class and three clases that inherit from that base class.

I've also imported another class that handles .conf files (stole it from the Update Manager project :) )

Still working on getting status updates written to the GUI in the middle of a series of statements, but that'll come.

In the mean-time, here's what the main window looks like now:
Multi_V2.jpg
That bar/frame just about the bottom row of buttons is for a Status Bar. I'm thinking I'll flip the position of the SB and the three buttons and make the SB visible only when there's something to be displayed there - which won't be very often.

Anyway, time will tell about how much longer this will take but I think I've rounded the (next to) last corner.
Scott
Quoting zerozero, "The usage of PPA's in debian-based
systems is risky at best and entails serious compatibility
problems; usually it's the best way to destroy an install"

Deleted User 2764

Re: Multi-Boot Application - Progress

Postby Deleted User 2764 » 07 Aug 2014 23:11

I assume "Copy" is the ISO to copy to the USB drive?

It all looks really good. Nice, neat and clean. Congrats on learning more Python. I'm still trying to get through the Intro to Linux course and hoping that maybe I really can finish by Aug. 15th. Then I want to try Python.

Keep working at it. And sometimes when you think you rounded one corner, you find another one that you get stuck on. Don't give up. Keep working it out. I learned never to say when I think I'll be done with a project because something always comes up that gets me baffled for a good amount of time. :lol:

User avatar
ScottQuier
Posts: 1781
Joined: 18 Jul 2013 15:55
Location: Newport News, VA

Re: Multi-Boot Application - Progress

Postby ScottQuier » 07 Aug 2014 23:16

RavenLX wrote:I assume "Copy" is the ISO to copy to the USB drive?

It all looks really good. Nice, neat and clean. Congrats on learning more Python. I'm still trying to get through the Intro to Linux course and hoping that maybe I really can finish by Aug. 15th. Then I want to try Python.

Keep working at it. And sometimes when you think you rounded one corner, you find another one that you get stuck on. Don't give up. Keep working it out. I learned never to say when I think I'll be done with a project because something always comes up that gets me baffled for a good amount of time. :lol:
The "Copy" will be re-labeled - It should have been "Add ISO" and the intent is "documented" in the screenshot found in the OP. Starting with a clean slate I re-worked the GUI from the ground-up. So, a couple of things are not labeled as they were and, in this case, that Radio Button should have been.

I know about the corners. I've been doing this work for almost 40 years. It's just the way python does OO that threw me a bit. But, I'm back on the horse and have fun!
Scott
Quoting zerozero, "The usage of PPA's in debian-based
systems is risky at best and entails serious compatibility
problems; usually it's the best way to destroy an install"


Return to “Code”

Who is online

Users browsing this forum: No registered users and 2 guests