====== 10. Semestral work - display and keyboard on PCIe bus ======
* for lecturer: [[..:..:..:internal:tutorials:10:start|tutorial 10]]
===== Task definition =====
Using the PCIe card EVB/Altera DB4CGX15 and the tool with LCD display and keyboard implement control program for a messaging system resembling Pager (predecessor of SMS). Each keyboard/Altera board is individual pager identified by unique ID in the system. The system is represented by standalone server in the computer network. The pager's ID and server's IP address are passed on command line. When started the pager is in the stand-by state. In this state every 5 seconds it checks if there are any incoming messages on the server. If there is a message, the pager beeps and displays the message. User has to confirm the message by a key. When the key is pressed, the pager sends "read" confirmation to the server and enters stand-by state.
The pager also allows to send a message to other pager in the system. While in stand-by mode, a user can press a designated key to the enter the messaging mode. First he/she has to enter other pager's ID and then enter the message. In easy version - the message consists of a numberic code up to six digits long, in advanced version the message is "text message" of unlimited length. If user do not push any button for 30 seconds, the pager discarts the message and enters stand-by mode.
The keys on the pager has to have acoustic feedback.
To request incoming and send outgoing messages from server you will use TCP/IP protocols. Simple server will be available later, but it's protocol is described later. This server allows numeric codes as messages for simple version. If you want to use advanced version for text messages, you have to implement your own. It will have to handle at least 10 unique pagers and at least 10 unread messages for each pager.
Your solution will consist of: the program, user documentation and technical documentation. On presenting your solution, teacher will check: completeness, functionality and your understanding of the program. The documentation is always mandatory, but you will get points, only if all other parts of your work is complete.
The deadline is on the last week (tutorial) of the semester. You will receive 5 points penalty of every started week after the deadline (starting on Monday midnight).
On presenting your semestral work we will check if your program works correctly and also if you understand your code and documentation. Documentation is always mandatory, but you will get points only if your program works and is complete. You can receive full points only if you present your semestral work before deadline. After that you will get a 5 point fine for every week passed.
==== Prefered steps and evaluation ====
* altorithm to find the PCIe card in system - equivalent to ''lspci -nn -v -d 1172:1f32'' (use ''/proc/bus/pci'') [1pt]
* map PCIe card registers to process memory and turn on the tool [1pt]
* bus cycle implementation and flashing with LEDs on the tool [2pts]
* LCD initiation and displaying text on the display [2pts]
* find button pushed and printing to the PC console
* keyboard scan [2pts]
* debouncing [1pt]
* key press accoustic feedback (using internal beeper) [1pt]
* entering pager's ID and displaying a message [2pts]
* completion - connecting the control algorithm with the tool (interface for LCD and controlling from tool keyboard) and communication with the server [max 3b]
* documentation in PDF [mandatory, in case of working solution max 2pts for user guide, 2pts for techical report]
===== Simple Pager Server Description =====
The simple server will be available in source code here. To compile the server, you must use ''make''.
{{:courses:a0b36apo:tutorials:10:apo-pagercentral-2015-1.tar.gz|}}
The following command can be used to run the server:
./pagercentral -v
The server by default listens on port 55556. You can verify your server using ''telnet'' program with following command:
telnet localhost 55556
To check new messages in the queue for a pager, you can use following command:
querrymessage 42
Response to this request contains
querrymessage_r
If there is no message in the queue, the '''' is text "none", otherwise it is ID of the message in the system, followed by actual code/text of the message. To check errors/inconsistencies in request/response check the console output.
To confirm that message was read, use:
confirmmessage 42 123456
Server then replies with ''confirmmessage_r'' followed by and .
Nová zpráva bude centrálnímu systému vyslané příkazem
sendmessage 42 53 1337
Odpověď bude místo textu zprávy obsahovat přiřazené unikátní číslo v rámci systému
sendmessage_r 42 53 123456
===== Hardware description =====
Hardware consists of input/output PCI card implementing memory mapped input/output gates and registers of the PCI address space and external hardware tool - matrix keyboard, LCD controller and 8 LED diods.
Card with the FPGA integrated circuit allows to set control and data bits on 8-bit parallel port. This port is used to generate signals on 8bit parallel bus. The card also contains 8-bit output register for data part of the bus (D0 - D7) and also 8-bit input register (used for results of read operation and also as a feedback for write operations).
Control register of the software controlled 8-bit bus.
^ Address on 8-bit bus ^ Direction ^ Abbreviation ^ Description ^
| 0 | WR | LCD_INST | Write instruction into LCD controller |
| 1 | RD | LCD_STAT | Read state register of LCD controller |
| 2 | WR | LCD_WDATA | Write data to display memory or character generator |
| 3 | RD | LCD_RDATA | Read data from LCD display memory |
| 1 | WR | LED_WR | Write to LED |
| 3 | WR | KBD_WR | Write to keyboard scan register |
| 0 | RD | KBD_RD | Read data from keyboard scan register |
Register programmed on PCIe card EVB/Altera [[http://www.ebv.com/en/products/categories/details/product/ebv-cyclone-iv-gx-development-kit.html|DB4CGX15]]
As you do not have root access to computers in room KN:E-328 you have to enable direct access to memory for read/write. To do this you have to run
''devmemrw''
^ PCI Bus address ^ Direction ^ Name ^ Description ^
| BAR0+0x0000 | RD/WR | RAM | Local memory 32kB RAM in FPGA |
| BAR0+0x8000 | RD/WR | timer_0 | Board build-in timer with IRQ |
| BAR0+0x8020 | WR | **emul_bus_data_out** | **Data signals - write data to set value on the bus** |
| BAR0+0x8040 | RD | **emul_bus_data_in** | **Data signals - read actual data set on the bus** |
| BAR0+0x8060 | WR | **emul_bus_addr** | **Address - address on the bus** |
| BAR0+0x8080 | WR | **emul_bus_ctrl** | **Control register of the bus** |
| BAR0+0x80A0 | WR | Led_Pio | Access to build-in LED on the PCIe card |
8-bit output control register with control signals of emulated bus (**emul_bus_ctrl**)
^ Bit ^ Signal name ^ Pin IO Modul ^ Pin of FPGA ^ Description ^
| 0 | RD | IO13 | L9 | Read command (active in L) |
| 1 | WR | IO14 | K9 | Write command (active in L) |
| 2 | | | | |
| 3 | | | | |
| 4 | | | | |
| 5 | | | | |
| 6 | CS0 | IO15 | N10 | Chip select (active in L) |
| 7 | PWR | IO18 | L11 | Turn on the tool |
8-bit output register connected to address wires of the simulated bus
^ Bit ^ Signal name ^ Pin IO Modul ^ Pin of FPGA ^ Description ^
| 0 | A0 | IO11 | K8 | Address bit 0 (LSB) |
| 1 | A1 | IO12 | N9 | Address bit 1 |
| 2 | | | | |
| 3 | | | | |
| 4 | | | | |
| 5 | | | | |
| 6 | | | | |
| 7 | | | | |
8-bit output register is connected to simulated data bus (**emul_bus_data_out**). The same mapping is for the input register (**emul_bus_data_in**).
^ Bit ^ Signal ^ Module IO Pin ^ FPGA Pin ^ Description ^
| 0 | D0 | IO1 | L4 | Data bus - bit 0 (LSB) |
| 1 | D1 | IO2 | M4 | Data bus - bit 1 |
| 2 | D2 | IO3 | N4 | Data bus - bit 2 |
| 3 | D3 | IO4 | N5 | Data bus - bit 3 |
| 4 | D4 | IO5 | L5 | Data bus - bit 4 |
| 5 | D5 | IO6 | N6 | Data bus - bit 5 |
| 6 | D6 | IO7 | M6 | Data bus - bit 6 |
| 7 | D7 | IO8 | L7 | Data bus - bit 7 (MSB) |
More control signals (no important for programmers).
IO10 (M9) = not WR and RD
= not IO14 and IO13
IO9 (N9) = (WR and RD and CS0) or not (WR or RD)
= (IO14 and IO13 and IO15) or not (IO14 or IO13)
Direction of the data flow (bus operation) is controlled by control register signals RD, WR, CS. The direction is set to output (from FPGA to bus) when (not WR and RD - pin ''IO14'' is set to ''1'' and pin ''IO13'' is set to ''0'').
Other signals of the FPGA are not relevant for your work.
IO19 (N12) = 0
IO20 (K10) = 1
===== Matrix keyboard =====
[[http://www.dribin.org/dave/keyboard/one_html/|Description how to determine pressed keys on a matrix keyboard]]
===== LCD display controller =====
LCD Controller is standard Hitachi HD44780.
To initialize LCD
* Write a constant CHMOD_LCD_MOD to address BUS_LCD_INST_o of the bus (for addresses table above).
* Wait at least 10 ms.
* Write again CHMOD_LCD_MOD to address BUS_LCD_INST_o of the bus.
* Wait at least 10 ms.
* Clear inner display memory by writing CHMOD_LCD_CLR on address BUS_LCD_INST_o of the bus.
* Turn on displaying a text from the display memory of a controller by write CHMOD_LCD_DON on address BUS_LCD_INST_o of the bus.
To write a character to the memory of the controller:
* Wait until bit CHMOD_LCD_BF is set to 0. The bit CHMOD_LCD_BF can be read from the bus address BUS_LCD_STAT_o.
* Write character value (eg 'A') on bus address BUS_LCD_WDATA_o.
To move cursor to the given position (the next character will be written to this place).
* Write a value CHMOD_LCD_POS+//// on the bus address BUS_LCD_INST_o. If you want to write to the second row, add 0x40 to the position.
More can be found for example in two PDFs available here [[http://ozirock.webs.com/hitachihd44780lcd.htm]]
===== Hits =====
===== First steps =====
The code below shows basic first steps you can start with. This code ''mmap''s the PCI device's memory space to program's address space and turns on and off the hardware tool. The code contains some hints and steps you should implement. To compile the code you can use ''make'' command. The ''make'' uses compile and linking instructions written in makefile. A sample Makefile that can be used to comple the code is in the archive below.
To compile a program using ''make'' command, first open a terminal application. Then use ''cd'' command to change directory to one with the makefile. And then use to ''make'' command (no parameters). This command compiles and links only changed files. If you want to comple whole project, you must remove all intermediate files using ''make clean''.
{{http://dcenet.felk.cvut.cz/elms-sdc/apo/zaklad.tar.gz}}
#define DEV_ENABLE "/sys/bus/pci/devices/0000:03:00.0/enable"
#define DEV_ADDRESS 0xfe8f0000
#define CTRL 0x8020
#include
#include
#include
#include
int pciEnable(int isEnable)
{
char cen = isEnable!=0 ? '1' : '0';
int enable = open(DEV_ENABLE,O_WRONLY); //TODO: find the path to DEV_ENABLE by searching through the list of PCI devices.
if(enable==-1) return 0; // TODO: Handle errors properly
write(enable,&cen,1); close(enable);
return 1;
}
void write_bus(uint8_t address, uint8_t data); //TODO write bus cycle
uint8_t read_bus(uint8_t address); //TODO read bus cycle
int main()
{
int soubor= open("/dev/mem",O_RDWR | O_SYNC);
if(soubor==-1) return 1; // TODO: Handle errors properly
// TODO: Replace DEV_ADDRESS constant and memory size (0x10000) by a search through the PCI devices
unsigned char * base = mmap(NULL,0x10000,PROT_WRITE | PROT_READ, MAP_SHARED,
soubor, DEV_ADDRESS);
if(base==MAP_FAILED) return 2; // TODO: Handle errors properly
if(pciEnable(1))
{
*(base+CTRL)=0xF0; // Turn on the power
sleep(1); // wait a second. For shorter delays use usleep (microseconds) or nanosleep (nanoseconds)
// TODO: Your semestral work goes here :).
*(base+CTRL)=0x00; // Turn off the power
}
pciEnable(0);
return 0;
}
//TODO: Podprogram pro zapis/cteni bytu na emulovane PCI sbernici
//+TODO: cteni klavesnice,
//+TODO zapis na LCD
//+TODO: program automatu na jizdenky
=== Testing connection to HW tool and function of the tool ===
{{courses:A0B36APO:tutorials:10:mmaped_8bit_bus_kbd-test-2015.tar.gz|}}
=== Programming ===
Following archive contains header files with display and keyboard register addresses on the emulated bus.
{{courses:A0B36APO:tutorials:10:mmaped_8bit_bus_kbd-headers-2015.tar.gz|}}
=== To browse directories in ''/proc/bus/pci'' ===
[[http://www.gnu.org/software/libc/manual/html_node/Opening-a-Directory.html#index-opendir-1413|opendir]]
[[http://www.gnu.org/software/libc/manual/html_node/Reading_002fClosing-Directory.html#index-readdir-1417|readdir]]
[[http://www.gnu.org/software/libc/manual/html_node/Reading_002fClosing-Directory.html#index-closedir-1421|closedir]]
=== Reading the PCI header ===
Files ''/proc/bus/pci/''//bb//''/''//dd//''.''//f// contains data from PCI headers (a copy of a PCI configuration space) from PCI devices found by OS.
[[http://www.gnu.org/software/libc/manual/html_node/Opening-Streams.html|fopen]]
[[http://www.gnu.org/software/libc/manual/html_node/Block-Input_002fOutput.html|fread]]
[[http://www.gnu.org/software/libc/manual/html_node/Closing-Streams.html|fclose]]
PCI Headrer structure is for example described on [[http://en.wikipedia.org/wiki/PCI_Configuration_Space|Wikipedii]]
Detailed description of PCI/PCIe bus architecture was given in lectures [[lectures:05:start|5. I/O podsystém 1]] and [[lectures:06:start|6. I/O podsystém 2]]. PCI/PCIe hierarchy is described mainly in following presentation: {{courses:A0B36APO:lectures:06:pci-bus-and-control-space.pdf|PCI a PCI Express sběrnice a PCIe}}.
In the first 4 bytes is encoded product and vendor identification - for our FPGA design is ventor ID set to **0x1172** and product ID is **0x1f32**.
Physical memory addresses for access are stored in BAR0 register. Lowest bits in register have to be masked (see eg. PCI BAR Bits table on Wikipedia).
To enable (turn on) the PCI device you must write "1" to file ''/sys/bus/pci/devices/0000:''//bb//'':''//dd//''.''//f///enable
=== Open and memory map ''/dev/mem'' ===
[[http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html|open]]
Use [[http://www.gnu.org/software/libc/manual/html_node/Operating-Modes.html#index-O_005fFSYNC-1359|O_FSYNC]] for direct access to avoid OS buffers and disk caches.
Physical address represented ''/dev/mem'' is the mapped to address space of the program and can be accessed in the same way as any other memory region. To map the memory use
[[http://www.gnu.org/software/libc/manual/html_node/Memory_002dmapped-I_002fO.html#index-mmap-1255|mmap]]
To avoid Copy on Write mechanism, you have to use MAP_SHARED option.
=== Access to card registers ===
To access board registers we will user pointer with type (volatile uint32_t *).
First we will try to read and write memory mapped to FPGA, then we will turn on the HW tool and simulate bus cycles.
Program for direct read and write memory mapped data registers from PCI device (registers have to be mapped via ''/dev/mem''). (It can also copy and fill memory regions).
{{courses:A0B36APO:tutorials:10:rdwrmem.tar.gz|}}
To use the program, you have to compile the source code using ''make''. After compiling the program you can use ''sudo ./rdwrmem'' to run the program. Please note the ''sudo'' - you have to be the superuser to run this program. Run ''sudo ./rdwrmem -h'' to display help and usage information.
./rdwrmem -h
Start address for write is specified by parameters ''-s''. To start experiments with our HW tool we demonstrate a work with **emul_bus_ctrl**. This register in on address ''BAR0 + 0x8040''. (BAR0 is the first Base Address Register). When you write a value with highest bit (bit 7) set to 1, the HW tool is switched on.
# Turn on power
./rdwrmem -s -F 0xFF
Now more difficult example - access to **KBD_WR** register and turn off the speaker.
# set up address of KBD_WR (address part of the bus)
./rdwrmem -s -F 0x03
# set up data for write (data part of the bus)
./rdwrmem -s -F 0x00
# Set control part of bus (activate WR signal)
./rdwrmem -s -F 0xFD
# Set control part of bus (activate WR signal, add CS signal)
./rdwrmem -s -F 0xBD
# wait - at least 10 microseconds
sleep 1
# deactivate CS0
./rdwrmem -s -F 0xFD
# deactivate WR
./rdwrmem -s -F 0xFF
===== Hardware technical documentation =====
==== Hardware tool / board of matrix keyboard MO_KBD3 documentation ====
{{courses:A0B36APO:tutorials:10:mo_kbd3-minimal-sch.pdf|mo_kbd3-minimal-sch.pdf - Simplified schematics of keyboard MO_KBD3}}
{{courses:A0B36APO:tutorials:10:mo_kbd3-sch.pdf|mo_kbd3-sch.pdf - Full schematics of MO_KBD3 including possible MCU connector for CAN bus access}}
{{courses:A0B36APO:tutorials:10:mo_kbd3-top.pdf|mo_kbd3-top.pdf - Components placement - upper side}}
{{courses:A0B36APO:tutorials:10:mo_kbd3-bot.pdf|mo_kbd3-bot.pdf - Components placement - lower side}}
==== Signal level conversion and interconnection board CON-DB4CGX15 documentation ====
{{courses:A0B36APO:tutorials:10:con-db4cgx15-sch.pdf|con-db4cgx15-sch.pdf - Schematics of interconnection board CON-DB4CGX15}}
{{courses:A0B36APO:tutorials:10:con-db4cgx15-top.pdf|con-db4cgx15-top.pdf - Components placement - upper side}}
{{courses:A0B36APO:tutorials:10:con-db4cgx15-bot.pdf|con-db4cgx15-bot.pdf - Components placement - lower side}}
===== Software emulated 8-bit bus interface FPGA design documentation (mmaped_8bit_bus) =====
{{courses:A0B36APO:tutorials:10:mmaped_8bit_bus-toplevel.pdf|mmaped_8bit_bus-toplevel.pdf - Top level design of PCIe subsystem, registers and FPGA pins interconnection}}
{{courses:A0B36APO:tutorials:10:mmaped_8bit_bus-sopc-avalon.pdf|mmaped_8bit_bus-sopc-avalon.pdf - Interconnection and mapping of the design registers to PCIe and Avalon bus in SW Altera SOPC Builder}}
[[tutorials:10:sopc_builder_pcie:start|Configuration of PCIe block in SW Altera SOPC Builder PCI Compiler]]
====== Virtual HW keyboard (APOHW) for QEMU Emulator ======
You can work on your semestral work even without access to HW. Virtual HW for APO course was implemented by Rostislav Lisový. The implementation is based on his diploma thesis [[http://rtime.felk.cvut.cz/hw/index.php/Humusoft_MF6xx|Education Material and Environment for PCI Driver Development for GNU/Linux]].
Detailed information about APOHW project can be found on [[http://rtime.felk.cvut.cz/hw/index.php/A0B36APO_Virtual_Hardware|APOHW virtual hardware implementation for QEMU]]
Emulation implements read/write to LED diods and write to the display. Keyboard input, LEDs state and display content is made accessible to user by provision of simple telnet server which listens on TCP port 55555. Next command can be used to monitor hardware state
''telnet localhost 55555''
Warning: Your semestral work have to work on physical HW. Virtual HW is mainly for testing purposes and to verify basic design of your work. APOHW does not implement timing requirements accurately.
Complete virtual system with APOHW implemented can be downloaded here:
{{courses:A0B36APO:tutorials:10:qemu-run-apohw-32.tar.gz|qemu-run-apohw-32.tar.gz}} (32-bit GNU/Linux verze)
{{courses:A0B36APO:tutorials:10:qemu-run-apohw-64.tar.gz|qemu-run-apohw-64.tar.gz}} (64-bit GNU/Linux verze)
It is implemented using QEMU with HW extension and contains Liux kernel, Busybox and some libraries. The executable is compiled for 32-bit Linux host system. If you have Ubuntu (or Debian) you have to install "qemu-system" and its depandencies (or its equivalents for other Linux systems).