Half assing AVR-DA without a Dragon

[T3sl4co1l]

So this thing exists,
https://github.com/mraardvark/pyupdi

But there doesn't seem to be a compiled/configured/ready-to-go version of it.  Guess I have to install Python (which I've managed to avoid for a surprisingly, perhaps embarrassingly, long time).

Ok, download 3.8 (last version supporting Win7), install it.  Installer didn't seem to set paths, or maybe it expects to be used from own current directory; which would be strange, but it's under AppData, so I guess it doesn't really matter?  Easy enough to set up myself.  Also set a path to Scripts, since I guess I need pip to work.

Run the pip command, it does indeed find the repo, puts it in /Lib/site-packages I see.  Ok.

Emits some warning about pip having a newer version.  Seems good.  Run pip update directly (note, it said to do it via python).  Hmm, the pip EXEs all updated, and all of them generate runtime errors.  Well that's not good.  Look up how to fix it.  Ah, run ensurepip.  Fixes it.  Run the correct python pip update command.  Seems fine.  Why would it be able to brick itself though?

Ok.  Have Python, have pyupdi installed.  Is it... just a module?  No, it doesn't execute.  Seems like I could put in a __main__ very easily myself, but is there some reason it's not there already?  Without knowing the reason I'd rather not screw with it.  How, then, to invoke it?  If I enter a path it'll just look for the path, not relative.  That's no good.  Surely I'm not supposed to python C:\Users\<me>\AppData\Local\Programs\Python\Python38\Lib\site-packages\updi\pyupdi.py every fucking time I want to run this thing.  Right?  ...Right?

Poke around a bit, find I can invoke it by python -m updi.pyupdi.  It emits a warning about finding it in sys.modules after import of package.  But it seems to run.  Safe to ignore?  No one else mentions having warnings.  Well, of the two or three websites I've found using this tool, no idea about chats/forums, those don't exist on the internet anymore. 👀 But they also don't mention anything else either, this is just automatic and expected...

Anyway, wired up a AVR64DA32, with the resistor and a MAX232 and USB serial dongle (I don't happen to have any TTL-level dongles around, oddly enough), running python -m updi.pyupdi -d avr64da32 -c <COM port> -i, from anywhere with the paths set up, seems to detect the device, yay (signature '1E9614' as expected).

Like, is this supposed to be this awkward?  Is it my fault for being on Windows?  (Yes, but still.  There's always some traditional working environment.  Did I just guess wrong, which kind here?)  If I was 10 again I have no doubt I'd understand the intricacies of a Python distro in no time, but alas it didn't exist back then, all I had was QBASIC to poison my mind..

Anyway, now onto the GCC toolchain...

Tim

[Siwastaja]

Completely normal python experience. You missed the usual step that nowhere it's described which language it is (there is no such thing as "Python"; there are Python2 and Python3) so you try to run it on the wrong Python, it kinda works, you kinda get all the gazillion libraries and dependencies installed after fixing a few cryptic errors, on that wrong Python, and finally you figure out that D'oh, should have used pip2/pip3/python2/python3 instead, and you download and install the correct Python and pip and start over, again fixing different cryptic errors and dependencies.

With all this, I wonder how Python seemingly is so popular, even among individuals I know are otherwise completely sane, rational and capable.

[T3sl4co1l]

Well, that would be a problem, I suppose.  Fortunately this specifies Python 3.

It's fantastically compact compared to most distributions: I've also got a copy of Cygwin64 (650MB), mingw32 (300MB), mingw64 (460MB), and assorted other stuff, so just POSIX/GCC environment(s) totaling 1.5GB.  I've got Ruby 2.7 (for one thing I tried out) taking a cool 1.1GB, and PHP 7.1 and 7.3 (for one other thing) totaling 320MB.

Of those, Ruby I know very little about, PHP7 isn't as much of an embarrassment as its earlier predecessors were (but, I think, still has some notorious cruft in the library functions?), and the GNU stuff is all over the place as usual (either it Just Works(TM), or you'll encounter a zillion inscrutable errors that only a kernel debugging could solve).  Python and PHP are the most compact by far, and as far as I know, the most usable, just sit down in an interpreter and play.  Copy some lines into a script file and save, now you have something to build on.

Software in general, my impression isn't that it's the fault of any particular distribution; most of them, that are extremely popular and well maintained, are that way for a reason, and are easy to use on any similarly mainstream system/configuration.  (Heck, MicroPython is even supported on modest MCUs.  It's no BASIC, but it's probably the closest thing today, and as powerful as one expects of languages these days.)  It's when you stray off that path, or use something less easy-to-use / well-documented, or something completely arbitrary like most any corporate codebase, that you get into the land of inscrutable bullshit.

Also expectation, to a certain extent.  Free projects are, by and large, put out there for convenience, and are worth precisely what you paid for it, if you know what I mean.  Like, the stuff I posted on Github, is only confirmed as meaningful to me, and maybe someone's found snippets, or the whole things, useful in some way, but it's also doubtful that anyone else has built the exact projects, without changes, drop in and go.  (Well, aside from the HTML+CSS+JS projects, that's kind of trivial, heh.)  Particularly where hardware is involved, and if that should differ, it could be a little bit to a huge pile of work to port it over.

So, that said, I expect Python to work, and it does seem to be quite easy to install, compact, and I'll likely give it some use for easy scripty things over the JS or other things I've been using.  The pyupdi thing, however, is just a personal project AFAIK, and clearly doesn't have any intention of being... necessarily usable, out of the box?  I mean the readme just barely says what it is, and says nothing about how to use it.  The author knows what they're doing, that's good enough for them, right?

Tim

[westfw]

is this supposed to be this awkward?

I don't think so.  When I tried it, I got a pyupdi.exe in
C:\Users\billw\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\Scripts\pyupdi.exe


That's a very weird place to put executable, and it wasn't in my path either, but if it were, a lot of the subsequent difficulties OP describe would not have happened.

(Windows in general seems to have a problem with "apps" that can install other applications.  You're lucky it doesn't consider python a a virus.)

[mraardvark]

Yes, Python has its issues...

at least you can drop pyupdi and just go for pymcuprog
if you have a py35-39 installed, and selected the "add scripts to path" when you installed, then you can just install using:
pip install pymcuprog
and then run it using:
pymcuprog --help

not _that_ bad, is it?

[fchk]

Microchip MPLABX plus a Snap or PICKIT4 does not work for you? It's all there, ready to use.

fchk

[T3sl4co1l]

Yes, Python has its issues...

at least you can drop pyupdi and just go for pymcuprog
if you have a py35-39 installed, and selected the "add scripts to path" when you installed, then you can just install using:
pip install pymcuprog
and then run it using:
pymcuprog --help

not _that_ bad, is it?

Hi, welcome!  Also, thanks!

Yeah but it's just a script, isn't that missing something?

Waaaaait, it grabbed an EXE too, put under Scripts.  Unexpected, but a welcome surprise.

Is pymcuprog the preferred, or better maintained or more usable or etc. package then?  I mean, it's mentioned on the README, but it's not exactly a firm instruction to go use it instead.

It's a shame that pyupdi is exactly what I'm searching for (i.e., something to run UPDI), so of course it shows up first in results...

Microchip MPLABX plus a Snap or PICKIT4 does not work for you? It's all there, ready to use.

I would've used AVR Studio but once they migrated to VS it bloated from a casual 500MB or whatever it was, to literally five gigs, two of which must be installed on the C: drive. WTF?  And MPLABX is just the evolution of that, then?

Tim

[oPossum]

MPLAB X is Netbeans, not VS. It is ~600 MB for the IDE. Compilers are separate.

There is also a standalone programming utility (MPLAB IPE) if you just want to do programming using a SNAP or PICkit4.

[T3sl4co1l]

Ok so I've got the toolchain building, or nearly so, albeit with -B and a jank ass path to the device-specs file that GCC doesn't know about by default (does it not just search everything under its path? I don't quite get it..).  Not sure if I just put these files in the wrong place again, or what.  (I could stand to upgrade from GCC 8.1 to, what's out now, 11?  And I see someone's forked and improved(?) to a libc3, though it still doesn't have includes for the latest devices.  But I kind of don't want to, as it'll probably break old projects.  Ugh, dependencies.)

So there's one thing remaining.  I see quite clearly, the specs file contains this line,

Code: [Select]

*link_data_start:
-Tdata 0x806000
and the linker generates this error,
ld.exe: address 0x806000 of Debug\DA_Test.elf section `.data' is not within region `data'
and, I don't think there's device-specific linker scripts or anything?  These are pretty simple devices relative to others, and in any case the datasheet clearly says SRAM starting at 0x4000, length 0x4000.

Also, from searching the label, it sounds like a linker script won't override this anyway.

So, uh, one: why is this line apparently wrong, and, is it okay to just change it?

Should I, Idunno, pull request to the MCP repo I downloaded it from?  Surely they know better...??!

Tim

[T3sl4co1l]

Scratch that, the datasheet diagram is misleading.  The model I'm using has SRAM at 0x6000 to 0x7fff.

So the spec is correct, and the link scripts are bad?

https://www.avrfreaks.net/comment/2905251#comment-2905251

This seems to suggest so.  And changing the .x* files (or, uh, most of them, I don't know which one it's using I just changed until it worked) as suggested, seems to fix it.

By god, it's not actually me this time?  What the hell!

Tim

[mraardvark]

Is pymcuprog the preferred, or better maintained or more usable or etc. package then?  I mean, it's mentioned on the README, but it's not exactly a firm instruction to go use it instead.

pymcuprog is maintained by Microchip, so
- new devices are released regularly
- it has more features
- its in a CICD system, so its known to work (in theory
- its released to pypi and github (snapshot)
The catch is that its actually intended primarily for using with EDBG/nEDBG/Atmel-ICE etc... but the pyupdi ("serialupdi") implementation slots in nicely inside the same wrapper and also get the same test coverage, almost for free.
 

[T3sl4co1l]

Sounds nice, but uh, is there any actual... documentation?

There's this, which looks to be "the" official thing,
https://microchip-pic-avr-tools.github.io/pymcuprog/index.html
and,
- Overview
right, I get that
- Library usage
fine for those that need it
- Support, Dependencies
okay sure but...
- Index
there's... there's command line parameters in here, right?  They're documented, right?

I would love to try it, I think I've got more or less the right magic incantation, but.. it doesn't even say the most basic things like what file formats it uses, or what literally any of the options do (beyond their obvious meanings, when applicable), let alone what values they can take (other than the -v and -H options, whatever those do, and wherever the "action"? word is supposed to fit in).

Like, just offhand, pyupdi takes some oddly verbose/literal fuse format, which I can write a converter for I suppose; while this... maybe takes binary?  I can't even tell, I literally have to read the source to puzzle it out.

Interestingly, flashing a bin file seems to work.  Bin fuse doesn't though.


On the pyupdi, speaking of -- it talks but doesn't program.  Maybe I fed it the wrong data (which... again...), but it just sits in an infinite loop while, I think merely maintaining keepalive on the bus (I'm watching UPDI on the scope, not for any particular purpose just out of curiosity).

Tim

[westfw]

USER documentation?  In a FILE?  That you can read before you install it?  Inconceivable!It does mention "pymcuprog --help", if you dig a bit...

Code: [Select]

pymcuprog --help
usage: pymcuprog [-h] [-d DEVICE] [-p PACKPATH] [-t TOOL] [-s SERIALNUMBER]
                 [-m MEMORY] [-o OFFSET] [-b BYTES] [-l LITERAL [LITERAL ...]]
                 [-f FILENAME] [-c CLK] [-u UART] [-i INTERFACE]
                 [-v {debug,info,warning,error,critical}] [-V] [-R] [--verify]
                 [-x]
                 [-H {tool-toggle-power,user-toggle-power,simple-unsafe-pulse}]
                 [-U] [-C]
                 {ping,erase,read,write,verify,getvoltage,getsupplyvoltage,reboot-debugger,setsupplyvoltage,getusbvoltage,reset}

Generic programmer of selected AVR, PIC and SAM devices

Basic actions:
    - ping: reads the device ID or signature
    - read: read NVM
    - write: write NVM
    - erase: erase NVM

positional arguments:
  {ping,erase,read,write,verify,getvoltage,getsupplyvoltage,reboot-debugger,setsupplyvoltage,getusbvoltage,reset}
                        action to perform

optional arguments:
  -h, --help            show this help message and exit
  -d DEVICE, --device DEVICE
                        device to program
  -p PACKPATH, --packpath PACKPATH
                        path to pack
  -t TOOL, --tool TOOL  tool to connect to
  -s SERIALNUMBER, --serialnumber SERIALNUMBER
                        USB serial number of the unit to use
  -m MEMORY, --memory MEMORY
                        memory area to access: all, 'calibration_row',
                        'config_words', 'eeprom', 'flash', 'fuses', 'icd',
                        'internal_sram', 'lockbits', 'signatures', 'user_id',
                        'user_row'
  -o OFFSET, --offset OFFSET
                        memory byte offset to access
  -b BYTES, --bytes BYTES
                        number of bytes to access
  -l LITERAL [LITERAL ...], --literal LITERAL [LITERAL ...]
                        literal values to write
  -f FILENAME, --filename FILENAME
                        file to write / read. Note that when reading to hex
                        file only eeprom, flash, fuses, config_words, user_row
                        memories will be written to the hex file
  -c CLK, --clk CLK     clock frequency in Hz (bps) for programming interface.
                        (eg: '-c 32768' or '-c 115k' or '-c 1M')
  -u UART, --uart UART  UART to use for UPDI
  -i INTERFACE, --interface INTERFACE
                        Interface to use
  -v {debug,info,warning,error,critical}, --verbose {debug,info,warning,error,critical}
                        Logging verbosity level
  -V, --version         Print pymcuprog version number and exit
  -R, --release-info    Print pymcuprog release details and exit
  --verify              verify after write from file
  -x, --timing          add timing output
  -H {tool-toggle-power,user-toggle-power,simple-unsafe-pulse}, --high-voltage {tool-toggle-power,user-toggle-power,simple-unsafe-pulse}
                        UPDI high-voltage activation mode
  -U, --user-row-locked-device
                        Writes the User Row on a locked device (UPDI devices
                        only)
  -C, --chip-erase-locked-device
                        Execute a Chip Erase on a locked device (UPDI devices
                        only)

Usage examples:

    Ping a device on-board a kit:
    - pymcuprog.py ping

    Ping a device using Atmel-ICE
    - pymcuprog.py -t atmelice -d atmega4809 -i updi ping

    Read the some bytes of flash:
    - pymcuprog.py read -m flash -o 0x80 -b 64

    Erase an UPDI device:
    - pymcuprog.py erase

    Erase a locked UPDI device:
    - pymcuprog.py erase --chip-erase-locked-device

    Set target supply voltage on a kit:
    - pymcuprog.py setsupplyvoltage -l 3.3

[westfw]

(the help examples are wrong, mostly.  At least with my install, you won't want to specify the ".py" on the command line.  Sigh.)
But .. it's GOOD, overall.  I wish more vendors would provide source code to program their chips via assorted programmers, that you could build your own apps around.  Even if the documentation IS a bit behind their "preferred vendor-proprietary tools.

[mraardvark]

You know, you could always log a bug (or even a feature request) on the github repo.
Or just complain amongst yourselves here...

[fchk]

Microchip MPLABX plus a Snap or PICKIT4 does not work for you? It's all there, ready to use.

I would've used AVR Studio but once they migrated to VS it bloated from a casual 500MB or whatever it was, to literally five gigs, two of which must be installed on the C: drive. WTF?  And MPLABX is just the evolution of that, then?

AVR Studio is from Atmel for AVR and got to Microchip by buying Atmel. Windows only It's basically a dead end, along with the former Atmel tools.

MPLABX is from Microchip and was originally deleloped for PICs. After the Atmel purchase it got support for Atmel chips - along with the PICKIT tools. PICKIT4 and ICD4 got two more IO lines in order to support the ATMEL chips in addition to PIC ICSP. This is where Microchip puts its efforts into.
MPLABX is multi-platform (Windows, Linux, MacOSX).

fchk

[westfw]

You know, you could always log a bug (or even a feature request) on the github repo.

Done:  https://github.com/microchip-pic-avr-tools/pymcuprog/issues/7

[T3sl4co1l]

USER documentation?  In a FILE?  That you can read before you install it?  Inconceivable!It does mention "pymcuprog --help", if you dig a bit...

Yeah I saw that, also on the docs.  This one (technically from a 3rd party, I think? but linked from the other page),
https://pypi.org/project/pymcuprog/
at least goes into a little actual detail about the options.  But still not a peep about file formats, no list of supported tools.  And an example is given with a .hex file, which seems suspicious as hell to me; does it actually check extension, then?  Or what?


So I've managed to get successful flashing, sometimes.

pyupdi is inconsistent.  It's not clear to me why, but sometimes it goes into an apparent loop.  This was confirmed with -v, at least as far as what I can see flipping past.  Though it looks different by the end, but I think I've already hit CTRL+C by then so I don't know if that's exit code or what.  Then it fails in a TypeError stack trace.

Here's the case when it fails out by itself:

pyupdi -d avr64da32 -c com11 -v -f "\Debug\DA_Test.hex"

Code: [Select]

INFO:phy send 5 bytes
INFO:phy data:  : [0x55, 0x8, 0x2, 0x10, 0x0]
INFO:phy receive : [0x3]
INFO:link LD from 0x001002
INFO:phy send 5 bytes
INFO:phy data:  : [0x55, 0x8, 0x2, 0x10, 0x0]
INFO:phy receive : [0x3]
INFO:link LD from 0x001002
INFO:phy send 5 bytes
INFO:phy data:  : [0x55, 0x8, 0x2, 0x10, 0x0]
INFO:phy receive : [0x0]
INFO:nvm Padding to blocksize 256 with 0xFF
INFO:nvm Paging into 256 byte blocks
INFO:nvm Writing page at 0x800000
Traceback (most recent call last):
  File "C:\Users\Tim Williams\AppData\Local\Programs\Python\Python38\Scripts\pyu
pdi-script.py", line 33, in <module>
    sys.exit(load_entry_point('pyupdi==0.0.0', 'console_scripts', 'pyupdi')())
  File "c:\users\tim williams\appdata\local\programs\python\python38\lib\site-pa
ckages\updi\pyupdi.py", line 111, in _main
    if not _process(nvm, args):
  File "c:\users\tim williams\appdata\local\programs\python\python38\lib\site-pa
ckages\updi\pyupdi.py", line 136, in _process
    return _flash_file(nvm, args.flash)
  File "c:\users\tim williams\appdata\local\programs\python\python38\lib\site-pa
ckages\updi\pyupdi.py", line 151, in _flash_file
    nvm.write_flash(start_address, data)
  File "c:\users\tim williams\appdata\local\programs\python\python38\lib\site-pa
ckages\updi\nvm.py", line 111, in write_flash
    return self._write_mem(address, data, self.device.flash_pagesize, use_word_a
ccess=True)
  File "c:\users\tim williams\appdata\local\programs\python\python38\lib\site-pa
ckages\updi\nvm.py", line 133, in _write_mem
    self.application.write_nvm(address, page, use_word_access=use_word_access)
TypeError: write_nvm_v1() got an unexpected keyword argument 'use_word_access'
INFO:phy Closing port 'com11'

It seems to erase, well enough:

pyupdi -d avr64da32 -c com11 -e -v
I may have to CTRL+C this (infinite? loop) then run it again, and that seems to make it work.

It seems like I can attempt a flash write, then erase, then a slightly different flash write (I've commented out some stuff in the program and compiled again to cause this change), and this causes it to loop.  (The "Device is locked" line appears more or less when I CTRL+C.)

Code: [Select]

...
INFO:phy send 2 bytes
INFO:phy data:  : [0x55, 0x8b]
INFO:phy receive : [0x82]
INFO:app Wait for NVMPROG
INFO:link LDCS from 0x0B
INFO:phy send 2 bytes
INFO:phy data:  : [0x55, 0x8b]
INFO:phy receive : [0x82]
INFO:app Wait for NVMPROG
INFO:link LDCS from 0x0B
INFO:phy send 2 bytes
INFO:phy data:  : [0x55, 0x8b]
Device is locked. Performing unlock with chip erase.
INFO:link Writing key
INFO:phy send 2 bytes
INFO:phy data:  : [0x55, 0xe0]
INFO:phy send 8 bytes
INFO:phy data:  : [0x65, 0x73, 0x61, 0x72, 0x45, 0x4d, 0x56, 0x4e]
INFO:link LDCS from 0x07
INFO:phy send 2 bytes
INFO:phy data:  : [0x55, 0x87]
INFO:phy receive : [0x8]
INFO:app Key status = 0x08
INFO:link LDCS from 0x0B
INFO:phy send 2 bytes
INFO:phy data:  : [0x55, 0x8b]
INFO:phy receive : [0x82]
INFO:app Entering NVM programming mode
INFO:link Writing key
INFO:phy send 2 bytes
INFO:phy data:  : [0x55, 0xe0]
INFO:phy send 8 bytes
INFO:phy data:  : [0x20, 0x67, 0x6f, 0x72, 0x50, 0x4d, 0x56, 0x4e]
INFO:link LDCS from 0x07
INFO:phy send 2 bytes
INFO:phy data:  : [0x55, 0x87]
INFO:phy receive : [0x18]
INFO:app Key status = 0x18
INFO:app Apply reset
INFO:link STCS 0x59 to 0x08
INFO:phy send 3 bytes
INFO:phy data:  : [0x55, 0xc8, 0x59]
INFO:app Check reset
INFO:link LDCS from 0x0B
INFO:phy send 2 bytes
INFO:phy data:  : [0x55, 0x8b]
INFO:phy receive : [0xa2]
INFO:app Release reset
INFO:link STCS 0x00 to 0x08
INFO:phy send 3 bytes
INFO:phy data:  : [0x55, 0xc8, 0x0]
INFO:app Wait for !reset
INFO:link LDCS from 0x0B
INFO:phy send 2 bytes
INFO:phy data:  : [0x55, 0x8b]
INFO:phy receive : [0x20]
INFO:app Wait for !reset
INFO:link LDCS from 0x0B
INFO:phy send 2 bytes
INFO:phy data:  : [0x55, 0x8b]
INFO:phy receive : [0x0]
INFO:link LDCS from 0x0B
INFO:phy send 2 bytes
INFO:phy data:  : [0x55, 0x8b]
INFO:phy receive : [0x0]
INFO:nvm Reading device info
INFO:phy send 2 bytes
INFO:phy data:  : [0x55, 0xe5]
INFO:app SIB read out as: bytearray(b'AVR     P:2D:1-3')
INFO:app Device family ID: 'AVR'
INFO:app NVM interface: 'P:2'
INFO:app Using PDI v2
INFO:link Using 24-bit updi
INFO:app Debug interface: 'D:1'
INFO:app PDI oscillator: '3MHz'
INFO:link LDCS from 0x00
INFO:phy send 2 bytes
INFO:phy data:  : [0x55, 0x80]
INFO:phy receive : [0x30]
INFO:app PDI revision = 0x3
INFO:link LDCS from 0x0B
INFO:phy send 2 bytes
INFO:phy data:  : [0x55, 0x8b]
INFO:phy receive : [0x8]
INFO:app Reading 3 bytes from 0x1100
INFO:link ST to ptr
INFO:phy send 5 bytes
INFO:phy data:  : [0x55, 0x6a, 0x0, 0x11, 0x0]
INFO:phy receive : [0x40]
INFO:link Repeat 3
INFO:phy send 3 bytes
INFO:phy data:  : [0x55, 0xa0, 0x2]
INFO:link LD8 from ptr++
INFO:phy send 2 bytes
INFO:phy data:  : [0x55, 0x24]
INFO:phy receive : [0x1e, 0x96, 0x14]
INFO:app Device ID = '1E9614'
INFO:app Reading 1 bytes from 0x0F01
INFO:link ST to ptr
INFO:phy send 5 bytes
INFO:phy data:  : [0x55, 0x6a, 0x1, 0xf, 0x0]
INFO:phy receive : [0x40]
INFO:link LD8 from ptr++
INFO:phy send 2 bytes
INFO:phy data:  : [0x55, 0x24]
INFO:phy receive : [0x13]
INFO:app Device rev = '1.3'
Device info: {'family': 'AVR', 'nvm': 'P:2', 'ocd': 'D:1', 'osc': '3', 'device_i
d': '1E9614', 'device_rev': '1.3'}
INFO:nvm Loading from hexfile 'Debug\DA_Test.hex'
INFO:nvm Loaded 8312 bytes from ihex starting at address 0x0000
INFO:nvm Adjusting offset to address 0x800000
INFO:app Chip erase using NVM CTRL
INFO:app Wait flash ready
INFO:link LD from 0x001002
INFO:phy send 5 bytes
INFO:phy data:  : [0x55, 0x8, 0x2, 0x10, 0x0]
INFO:phy receive : [0x0]
INFO:app NVMCMD 32 executing
INFO:link ST to 0x001000
INFO:phy send 5 bytes
INFO:phy data:  : [0x55, 0x48, 0x0, 0x10, 0x0]
INFO:phy receive : [0x40]
INFO:phy send 1 bytes
INFO:phy data:  : [0x20]
INFO:phy receive : [0x40]
INFO:app Wait flash ready
INFO:link LD from 0x001002
INFO:phy send 5 bytes
INFO:phy data:  : [0x55, 0x8, 0x2, 0x10, 0x0]
INFO:phy receive : [0x3]
INFO:link LD from 0x001002
INFO:phy send 5 bytes
INFO:phy data:  : [0x55, 0x8, 0x2, 0x10, 0x0]
INFO:phy receive : [0x3]
INFO:link LD from 0x001002
INFO:phy send 5 bytes
INFO:phy data:  : [0x55, 0x8, 0x2, 0x10, 0x0]
INFO:phy receive : [0x3]
INFO:link LD from 0x001002
INFO:phy send 5 bytes
INFO:phy data:  : [0x55, 0x8, 0x2, 0x10, 0x0]
INFO:phy receive : [0x0]
INFO:nvm Padding to blocksize 256 with 0xFF
INFO:nvm Paging into 256 byte blocks
INFO:nvm Writing page at 0x800000
Traceback (most recent call last):
  File "C:\Users\Tim Williams\AppData\Local\Programs\Python\Python38\Scripts\pyu
pdi-script.py", line 33, in <module>
    sys.exit(load_entry_point('pyupdi==0.0.0', 'console_scripts', 'pyupdi')())
  File "c:\users\tim williams\appdata\local\programs\python\python38\lib\site-pa
ckages\updi\pyupdi.py", line 111, in _main
    if not _process(nvm, args):
  File "c:\users\tim williams\appdata\local\programs\python\python38\lib\site-pa
ckages\updi\pyupdi.py", line 136, in _process
    return _flash_file(nvm, args.flash)
  File "c:\users\tim williams\appdata\local\programs\python\python38\lib\site-pa
ckages\updi\pyupdi.py", line 151, in _flash_file
    nvm.write_flash(start_address, data)
  File "c:\users\tim williams\appdata\local\programs\python\python38\lib\site-pa
ckages\updi\nvm.py", line 111, in write_flash
    return self._write_mem(address, data, self.device.flash_pagesize, use_word_a
ccess=True)
  File "c:\users\tim williams\appdata\local\programs\python\python38\lib\site-pa
ckages\updi\nvm.py", line 133, in _write_mem
    self.application.write_nvm(address, page, use_word_access=use_word_access)
TypeError: write_nvm_v1() got an unexpected keyword argument 'use_word_access'
INFO:phy Closing port 'com11'



With pymcuprog, it seems more consistent in operation, but inconsistent in result.  I think I've determined the device must be erased prior to flashing.  Does it... not just do this automatically?  Even when verifying?  (Mind, I don't think --verify actually changes anything about execution, seems just adding the extra step at the end.  Just covering for the possibility that it could.  Again, I don't know what's in this thing yet.)  And there's only space for the one "action", so I have to run two separate commands just to get actual working code on a thing?  And no, this particular issue isn't a semantic problem, there is a way to get there -- I can toss that in a batch file, fine, so be it.  But surely this one thing, this one little thing, must be so blatantly obvious, how could it possibly be overlooked?  I mean, I'd love to have something that just drops in, like, avrdude I've never had trouble with (though I do have an official, if dated, dongle for it (AVRISPMKII), no hackery involved there).

Still not sure what I need to do fuses.  It tells me,

Code: [Select]

pymcuprog.pymcuprog_main - ERROR - Operation failed with IndexError: bytearray index out of range
when I specify a binary file, or

Code: [Select]

pymcuprog.pymcuprog_main - ERROR - Operation failed with ValueError: 44 bytes of data at offset 0 is outside the boundaries of 'fuses' with size 16
when giving hex (looks like it's reading it literally in this case, not checking if it's hex; inconsistent?).

I really don't know what it thinks it's doing as the device only has the 9 bytes of fuses.  And even then, of which two don't exist.

Tim

[Nominal Animal]

With all this, I wonder how Python seemingly is so popular, even among individuals I know are otherwise completely sane, rational and capable.

Hey! 

In my defense: on the Linux distributions I use, Python and most Python libraries (well, all that I use, anyway) are provided via the package manager and not PyPi, and are properly versioned, so all that mess is blissfully completely avoided.  The only thing is to remember to use python3/pydoc3 instead of python/pydoc when using Python3.

So no, I would not accept that kind of mess in using Python-based tools myself, either.

I do prefer C for this kind of system-level tool, and usually use either avrdude (for non-Teensy AVRs) or Teensy Loader (the GUI version provided in Teensyduino, or the command-line GPL 3 version) for programming my microcontrollers; and didn't know these Python-based versions even existed.
I also use shell scripts to launch Python applications by explictly invoking the interpreter, and recommend doing so, when asked.

(Using exec python3 ... in a Bash/dash/sh shell script the shell process is replaced by the python interpreter, so no extra shell remains in memory while the Python code runs.  If the Python environment requires additions/changes to work on a specific distribution or even user, it is trivial to do that in the shell script.  Another variant I sometimes use in such a launcher shell script is ( setsid python3 ... & ) </dev/null 2>/dev/null >/dev/null which starts the Python in a new session, detached from the current controlling terminal, and in the "background": the shell script returns immediately, but the Python code continues running, and is not killed even when the terminal it was started from is closed.  No extra shell process then either.  Very useful if you want to run stuff over e.g. SSH that won't get killed if you exit that particular SSH connection, or want to launch GUI applications from a terminal.)

[westfw]

I wonder how Python seemingly is so popular

Well, because of stuff like:

Code: [Select]

        from intelhex import IntelHex
        ih = IntelHex()
        ih.loadhex(filename)
        data = ih.tobinarray()
("That was easy!")

[nctnico]

I wonder how Python seemingly is so popular

Well, because of stuff like:

Code: [Select]

        from intelhex import IntelHex
        ih = IntelHex()
        ih.loadhex(filename)
        data = ih.tobinarray()
("That was easy!")

For the programmer maybe but for someone who wants to install the program getting the matching library versions it is a hell. One of my customers has a program developed in Python but getting it installed takes a good part of a day (and no guarantee it actually can be installed) due to the many Python libraries it needs which may not be compatible with the current Linux distributions. I very much regret not pushing that customer to just write the software in C++ and link it statically to the libraries it needs.  For a reasonably complex Python program the easiest way is to distribute it as a VM image 

[Nominal Animal]

One of my customers has a program developed in Python but getting it installed takes a good part of a day (and no guarantee it actually can be installed) due to the many Python libraries it needs which may not be compatible with the current Linux distributions.

Assuming distributing it is not possible as a .deb and/or .rpm package with the Python libraries as dependencies, the next best solution is to include the libraries it needs as a set, in os-arch/lib/pythonversion/ relative to suitable base directory, and use a wrapper script to execute a Python interpreter.  The wrapper sets PYTHONHOME and any other PYTHON* environment variables needed to use the included modules instead of system ones when executing the Python code.  It can be useful to include the Python interpreter binary itself, too, making it self-contained.

It's not really a Python issue, nor a Linux issue.  These people just do not know how to correctly/practically distribute stuff.  Instead, they make a half-assed tarball or ZIP of their script files, and tell others to copy those somewhere "and they'll hopefully work"...  Horrible pain ensues, with "worksforme" and shrugs from the original devs.

I've often wondered how to help with this.  Putting stuff on the net isn't going to make a difference, unless that stuff is promoted by Authority Figures, but playing that sort of social games pisses me off.

[nctnico]

One of my customers has a program developed in Python but getting it installed takes a good part of a day (and no guarantee it actually can be installed) due to the many Python libraries it needs which may not be compatible with the current Linux distributions.

Assuming distributing it is not possible as a .deb and/or .rpm package with the Python libraries as dependencies, the next best solution is to include the libraries it needs as a set, in os-arch/lib/pythonversion/ relative to suitable base directory, and use a wrapper script to execute a Python interpreter.  The wrapper sets PYTHONHOME and any other PYTHON* environment variables needed to use the included modules instead of system ones when executing the Python code.  It can be useful to include the Python interpreter binary itself, too, making it self-contained.

It is not that simple. There are also incompatibilities between system libraries and Python packages so you'd have to include all the system libraries that the Python packages use as well. Most commercial Linux software already does that but it is a pain to assemble such a software package. A statically linked executable OTOH is easy to distribute.

[Nominal Animal]

One of my customers has a program developed in Python but getting it installed takes a good part of a day (and no guarantee it actually can be installed) due to the many Python libraries it needs which may not be compatible with the current Linux distributions.

Assuming distributing it is not possible as a .deb and/or .rpm package with the Python libraries as dependencies, the next best solution is to include the libraries it needs as a set, in os-arch/lib/pythonversion/ relative to suitable base directory, and use a wrapper script to execute a Python interpreter.  The wrapper sets PYTHONHOME and any other PYTHON* environment variables needed to use the included modules instead of system ones when executing the Python code.  It can be useful to include the Python interpreter binary itself, too, making it self-contained.

It is not that simple. There are also incompatibilities between system libraries and Python packages so you'd have to include all the system libraries that the Python packages use as well.

Yes, I did mean including the native libraries the Python packages (and Python interpreter itself – expat, zlib, and standard C library).

Most commercial Linux software already does that but it is a pain to assemble such a software package. A statically linked executable OTOH is easy to distribute.

I disagree with "pain", but yes: statically linked binaries are easy.

You see, I use a set of scripts that use ldd to track the full set of native libraries, recursively.  (Since I currently use Debian derivatives, I also do dpkg-query -S path-to-library to obtain the set of packages they correspond to.)  This handles the dependencies among native libraries.

The ones it does NOT discover, are those that are loaded at run time via libdl (<dlfcn.h>), i.e. dlopen() and dlsym().

Another approach that does detect those too, and works better for Python stuff, is to run the application under strace, specifically strace -f -y -e open,openat -o logfile python3 application [args...], logging everything the process or its children open to logfile.  The issue in this approach is that one must then perform typical actions in the application to be sure to capture all libraries accessed during a normal session; and tends to involve more human effort than the above.  So, effort yes, but no pain/annoyment per se, in my opinion.

(Basically another script takes the logfile, determines which of those opened files are libraries and which Python files, and uses ldd determine the set of native libraries needed.  Many of the opened files are extra, including translations and even some native libraries like the resolver. The Python files list is then compared against the documentation or source, to see if the logged test use covered all the Python modules mentioned in the source; I let a human do this part.)

For native libraries in Linux, one can set the LD_LIBRARY_PATH environment variable to set the (primary) location where libraries are looked at.

So, if you start with a skeleton where the wrapper script is /usr/bin/appname, and the rest of the application resides in say /usr/lib/appname/ or /usr/share/appname with a Python-compatible subtree under there (lib/pythonversion/) and libraries in say os-arch/lib/, it really should not be painful; you should be able to do this in less than a day for a typical Python application.  The os-arch prefix (obtained using say uname -sm | tr ' A-Z' '-a-z') lets the exact same package work across not just Linux distributions on different hardware, but on *BSD variants as well.
(Obviously, the "installer" could omit the unneeded os-arch subtrees, to save disk space.)
Yes, it is not trivial, and it does take time, but packaging a project always takes some time, unless you explicitly keep track of the dependencies during development and automate (for the current OS and hardware architecture) the project-packaging using that dependency list.

In other words, what I am saying, is that the packaging does not have to be annoying/painful.  If you (plural you, meaning anyone who has distributed software) haven't done this before, I recommend trying it this way (using strace, and then ldd recursively on the native libraries to determine the full set of native libraries needed) before condemning it as too painful or requiring too much effort to consider.

[westfw]

getting it installed takes a good part of a day (and no guarantee it actually can be installed) due to the many Python libraries it needs which may not be compatible with the current Linux distributions.

I hate to tell you, but I have the same problem with C and C++ code (and presumably other languages.)  "dependency hell."  Especially if you have a host environment that is somewhat behind the latest OSSW tools - life got a lot easier for me after I figured out that I should have a separate build environment (gcc and libraries) for OSSW that I wanted to install (with various package managers), vs the "standard build environment" that got installed with Apple's XCode environment :-(  (I'm pretty sure a lot of it was bogus - automake or whatever claiming that it needed "version N" of some library because that was what the OP built it with, when practically any version WOULD have worked.  But I couldn't figure it our.)

At least with Python, there's more chance that it will work on Mac or PC even if the OP used linux (or ... switch those around.)