Serial port communication (Modbus)

I'm trying to communicate with yaMBSiavr running on an ATMega328 in SimulIDE 1.0.0-RC3 via a serial port, but it seems the serial communication is not working how I expect. I use two linked virtual serial ports (I have tried using both socat and tty0tty for this), one of which I connect to with SimulIDE and the other with a simple Python script that sends a Modbus command. yaMBSiavr does not respond to this message; a breakpoint indicates that it fails a format check and is ignored by yaMBSiavr. When I look at the incoming message in the serial monitor on the simulated AVR it's not what I expect: I'm sending the bytearray [0x01,0x03,0x00,0x00,0x00,0x01,0x84], but serial monitor shows 65 16 64 65 252 (in decimal I would expect 1 3 0 0 0 1 132). It looks like it's being converted to ASCII characters? I've probably misunderstood something, but what?

Hi Lomax.

As I understand the data transmitted by the atmega is already wrong .
How is the Usart configured in the atmega328? (values of UCSR0A, B and C)

Hello,

I haven't got to the point where the AVR sends any response; it is running as a Modbus slave and only transmits when asked to do so by the Modbus master (in this case my Python script). The AVR Modbus slave application never recognises any incoming messages, and thus never responds. The USART registers are set as follows:

UCSR0A 0010 0000
UCSR0B 1001 1000
UCSR0C 0000 0110

I have now tested with a completely different Modbus application on the AVR called liblightmodbus, which is much larger and more capable than yaMBSiavr, but I still cannot get the AVR to respond to any incoming messages.

Edit: To clarify, I can see that the AVR detects incoming messages by setting a breakpoint in the interrupt handler, but it seems the messages are not recognised as valid Modbus commands by the firmware on the AVR so no further action is taken.

I haven't got to the point where the AVR sends any response; it is running as a Modbus slave and only transmits when asked to do so by the Modbus master (in this case my Python script).

Ok, now I understand.

Be sure that the Serial Port component is configured at the same baudrate than the AVR, and 8 bits, 1 stop bit.
You can also open a serial monitor in the Serial Port component, and see what is sending to the AVR, to see if it is receiving the correct data from the virtual serial port and sending it to the AVR.

Thanks - I've already checked that the serial port settings are correct, and I'm using the serial monitor on the AVR rather than the serial port, so I'm looking at the data it receives. I suspect it's the serial monitor that does the conversion to ASCII, and that the AVR is receiving the correct values, but I'm not 100% sure. It would be helpful if the serial monitor had a raw decimal and/or hex display mode as well - some serial communication protocols (like Modbus RTU) don't make sense in ASCII...

I'm using the serial monitor on the AVR rather than the serial port, so I'm looking at the data it receives.

Yes, but you don't know where is the problem, maybe the AVR is receiving the correct data but there is some error or maybe that there is an error in Serial Port or maybe Serial Port is already receiving the wrong data...
You should open both to have an idea of what is exactly happening.

I suspect it's the serial monitor that does the conversion to ASCII, and that the AVR is receiving the correct values, but I'm not 100% sure.

You are watching raw decimal values (Print: Value).

arcachofo wrote:
Yes, but you don't know where is the problem, maybe the AVR is receiving the correct data but there is some error or maybe that there is an error in Serial Port or maybe Serial Port is already receiving the wrong data...
You should open both to have an idea of what is exactly happening.

The data in the serial monitor of the serial port looks the same.

arcachofo wrote:
You are watching raw decimal values (Print: Value).

This is why I posted here: the decimal values shown are incorrect. Look at my first post.

Edit: to clarify, when I send the value 0x84 (132 in decimal), the serial monitor shows 252 instead. When I send the value 0x01 the serial monitor shows 65. And so on.

Edit2: The Python script that sends the data could not be simpler:

Code:

import serial

msg = bytearray([0x01,0x03,0x00,0x00,0x00,0x01,0x84])
port = serial.Serial("/dev/tnt0")
port.baudrate = 9600
port.bytesize = 8
port.parity = serial.PARITY_NONE
port.stopbits = 1

port.write(msg)


This should be interpreted by the AVR firmware as the Modbus command "read register #0".

Lomax wrote:When I send the value 0x01 the serial monitor shows 65.

One way I can see this happening is if something is trying to interpret the value as ASCII: 0x01 is the "NULL" control character, which some applications will show as ^A and "A" is 65 in ASCII decimal. But I have no idea what is performing this conversion - I was hoping you might It's very confusing. I have built many applications which communicate over serial protocols, including Modbus, but never seen anything like this.

The data in the serial monitor of the serial port looks the same.

So the serial port is already receiving the wrong data.
This seems like a problem before reaching SimulIDE.

One way I can see this happening is if something is trying to interpret the value as ASCII: 0x01 is the "NULL" control character, which some applications will show as ^A and "A" is 65 in ASCII decimal. But I have no idea what is performing this conversion - I was hoping you might Wink It's very confusing. I have built many applications which communicate over serial protocols, including Modbus, but never seen anything like this.

In general is quite simple, nor the serial port or the serial monitor is performing any conversion.
Serial Port just receive and send data and Serial Monitor shows that data either in ASCII format or decimal value.

I did some test sending data from minicom and it is working for me (everything configured as 9600 8N1).
I'm using socat like this:
sudo socat PTY,link=/dev/ttyS1,mode=0666 PTY,link=/dev/ttyS2,mode=0666

Code:

import serial

msg = bytearray([0x01,0x03,0x00,0x00,0x00,0x01,0x84])
port = serial.Serial("/dev/tnt0")
port.baudrate = 9600
port.bytesize = 8
port.parity = serial.PARITY_NONE
port.stopbits = 1

port.write(msg)

Many years I don't use Python for Serial communication, but if I remember well you need to open the serial port before writing to it: port.open()

arcachofo wrote:
So the serial port is already receiving the wrong data.
This seems like a problem before reaching SimulIDE.

Capturing the data sent to the virtual serial port with cat and viewing the result as hex shows clearly that the correct data is being sent to SimulIDE:

Code:


$ cat /dev/tnt1 > capture.txt
^C
$ xxd capture.txt
00000000: 0103 0000 0001 84                        .......


How or why this ends up being interpreted as 65 16 64 65 252 (or A<0x10>@Aü) I still have no idea.

Code:

$ cat /dev/tnt1 > capture.txt
^C
$ xxd capture.txt
00000000: 0103 0000 0001 84 

If I understand correctly that means that the data has not been readen yet?
If the data was readen it would be empty...

arcachofo wrote:
I did some test sending data from minicom and it is working for me (everything configured as 9600 8N1).
I'm using socat like this:
sudo socat PTY,link=/dev/ttyS1,mode=0666 PTY,link=/dev/ttyS2,mode=0666

How did you get Minicom (a terminal emulator) to send raw bytes? If I send an ASCII string to SimulIDE from Python it appears correctly in the serial monitor, but Modbus commands can contain non-ASCII values beyond 127 and in the control character range 0-31.

And besides that the mystery remains: why does the byte 0x84 (for example) show up in serial monitor as 252 when the decimal value of that byte is 132? Again, I can see that the correct values are being received on the virtual serial port /dev/tnt1, so I still think the problem is in SimulIDE.

arcachofo wrote:If I understand correctly that means that the data has not been readen yet?
If the data was readen it would be empty...

It's just a test, that I performed in order to verify that the correct data is being received at /dev/tnt1. Not that I ever doubted this - I'm just trying to be thorough.

How did you get Minicom (a terminal emulator) to send raw bytes?

Any application using serial always send raw bytes, in this case it sends the ASCII code of a character as a raw byte.
It doesn't matter if you enter the value directly or you enter a character.
Anyway I tested with another program sending raw values, but that is just irrelevant.

If I send an ASCII string to SimulIDE from Python it appears correctly in the serial monitor, but Modbus commands can contain non-ASCII values beyond 127 and in the control character range 0-31.

That seems like a problem in the application that sends the data.
Because for Simulide it does not matter if the value received is printable or if it's beyond 127, it just receives a byte, if you select to print as Value it will show the raw value received, if you select ASCII it will print something or not, but the value received is the same and SimulIDE works the same for any value.

And besides that the mystery remains: why does the byte 0x84 (for example) show up in serial monitor as 252 when the decimal value of that byte is 132?

Maybe because it is receiving 252.

Again, I can see that the correct values are being received on the virtual serial port /dev/tnt1, so I still think the problem is in SimulIDE.

Could be, but that test was not performed while communicating with SimulIDE...
And all the tests I do are working correctly.

It is also weird that you are sending 7 bytes and receiving only 5.
Did you test by connecting to a serial terminal and sending data manually?

Btw... did you see this message? https://simulide.forumotion.com/t1287-serial-port-communication-modbus#5626

Here testing with minicom:


And here with another sending raw values:

I'm sorry, I think you are mistaking me for someone else. Unlike that person I do have some experience of serial communications on both the software and hardware side. What I don't have is a lot of experience using SimulIDE - in fact I am completely new to it. I need your help understanding how SimulIDE works, how to interpret what I'm seeing, and figuring out why it's behaving how it does. I don't think I need any help writing Python software that sends and receives serial data, configuring virtual serial ports or testing that this combination behaves how it should. The problems I'm having - whether the consequence of my inexperience, a confusing UI, AVR misconfiguration or an actual bug in SimulIDE - start and end inside SimulIDE.

arcachofo wrote:Any application using serial always send raw bytes, in this case it sends the ASCII code of a character as a raw byte. It doesn't matter if you enter the value directly or you enter a character.

Minicom is a terminal emulator and as such mainly deals with sending and receiving encoded characters. It falls well short if you want to send the hex value 0x84, or binary 1011 1011, and does not know what to do when receiving non-ASCII/ANSI/VT-whatever data. Terminal emulators are intended for human <> machine and human <> human communication and are not really suitable for debugging machine <> machine communication, where much of the data cannot reasonably be encoded to printable characters. I know this because I have been using terminal emulators like Minicom since 1989. Python by contrast is great for debugging machine <> machine serial communication, since it makes it easy to convert between hex/binary/ASCII/UTF-8 and further process the data, whether for display or computation.

arcachofo wrote:Anyway I tested with another program sending raw values, but that is just irrelevant.

Here we are in complete agreement.

arcachofo wrote:And all the tests I do are working correctly.

I don't see how any test other than running my simulation would prove anything.

arcachofo wrote:It is also weird that you are sending 7 bytes and receiving only 5.

Yes. Yes it is. Maybe I should ask a question in the forum about what might be causing this...

arcachofo wrote:Did you test by connecting to a serial terminal and sending data manually?

I understand you like using text terminal software for this, and that's ok, to each their own. Yes, as I've stated previously I did try sending ASCII data and it did show up correctly in the serial port's serial monitor. I've just checked now though, and it is garbled in the AVR's serial monitor.

Code:


import serial

port = serial.Serial("/tmp/ptyIN")
port.baudrate = 9600
port.bytesize = 8
port.parity = serial.PARITY_NONE
port.stopbits = 1

msg = "Hello World!"
port.write(msg.encode("ascii"))


arcachofo wrote:Btw... did you see this message? https://simulide.forumotion.com/t1287-serial-port-communication-modbus#5626

Of course, but I saw no reason to waste time on it, because it's both incorrect and beside the point. The problem I'm having is not related to sending data to SimulIDE, I already confirmed that this works and told you about it earlier in the thread:

Lomax wrote:If I send an ASCII string to SimulIDE from Python it appears correctly in the serial monitor

Writing this made me look again though, and I may have made a mistake in watching only the AVR's serial monitor, because I can see that the correct values are appearing in the serial port's serial monitor - but they are still incorrect in the AVR's serial monitor:

Code:


import serial

port = serial.Serial("/tmp/ptyIN")
port.baudrate = 9600
port.bytesize = 8
port.parity = serial.PARITY_NONE
port.stopbits = 1

msg = bytearray([0x01,0x03,0x00,0x00,0x00,0x01,0x84])
port.write(msg)


Maybe this observation will allow us to make progress?

Ha. I think I found the problem. Although both the Modbus libraries I tried running on the AVR allow setting the baudrate, it seems I also had to change the clock frequency of the AVR. After setting it to 16 MHz I am now seeing the correct values on the AVR's serial monitor:



I knew that serial communication on the AVR depends on precise timing, and that clock frequency is a factor in determining baud rate, but assumed this was handled by the Modbus libraries - in fact I even looked at the calculation performed by one of them for this. I also did not know that the serial monitor on the AVR showed the data from behind the USART, though that makes perfect sense now.

Lomax wrote:I'm sorry, I think you are mistaking me for someone else. Unlike that person I do have some experience of serial communications on both the software and hardware side.

To say it politely: I don't mind who you are or what is your experience.
But the fact that you are continuously talking about it instead of checking for errors in what you do, tells me something about you.

For example:

Lomax wrote:

arcachofo wrote:
Yes, but you don't know where is the problem, maybe the AVR is receiving the correct data but there is some error or maybe that there is an error in Serial Port or maybe Serial Port is already receiving the wrong data...
You should open both to have an idea of what is exactly happening.

The data in the serial monitor of the serial port looks the same.

Looks the same or you didn't even try?

Minicom is a terminal emulator and as such mainly deals with sending and receiving encoded characters. It falls well short if you want to send the hex value 0x84, or binary 1011 1011, and does not know what to do when receiving non-ASCII/ANSI/VT-whatever data. Terminal emulators are intended for human <> machine and human <> human communication and are not really suitable for debugging machine <> machine communication, where much of the data cannot reasonably be encoded to printable characters. I know this because I have been using terminal emulators like Minicom since 1989. Python by contrast is great for debugging machine <> machine serial communication, since it makes it easy to convert between hex/binary/ASCII/UTF-8 and further process the data, whether for display or computation.

All this discourse is irrelevant because I was using it to send data, not to receive it.
And I did it because you said that the serial port was receiving the wrong data.

Anyway I'm happy that FINALLY you actually opened the monitor in the serial port instead of talking about your incredible skills in computer science and communications.

Ooops, you lost me.

What I lost is a lot of time testing and reading and writing about something that was not real.