Integrate with Ikea Trådfri Gateway as bridge to communicate and control connected Ikea's ZigBee based smart lights, switches, and sensors

Ikea have just released a new app-controlled network-attached home automation hub which will serve as a Gateway to control its new "Trådfri" series of affordable smart lights / lightbulbs, switches / remotes, and sensors, which in turn so far all uses ZigBee based protocols. These products are set to be released on the 31st of March 2017 in selected countries around the world.
https://www.cnet.com/news/ikeas-rolling-out-a-brand-new-smart-home-lineup/
http://www.ikea.com/ms/sv_SE/customer-service/about-our-products/smart-lighting/index.html
"Trådfri" means 'wireless' in Swedish, and Ikea have so far announced this very aggresivly low-priced network-attached (Ethernet) "Ikea Trådfri Gateway" home automation hub in their "Tradfri" series, as well as a  wireless Motion Sensor Kit (that have integrated light sensor too), a wireless Dimmer Remote (which is accelerator-based), a wireless multi-switch remote, and several smart light bulbs of different formats and even a few unique panel lights. All these products will then be released in most other contries worldwide too as Ikea steps up manufacturing (and irons our the initial software bugs I guess).
Ikea had already leaked news about this upcoming gateway/hub more than 6-months ago, during the summer or 2016, and at that time they also revealved that they will use ZigBee and keep validated access to the gateway/hub as open as possible, including providing an open API for this network-attached home automation hub.
Ikea in Sweden are first to post this news about the network-connected Home Automation Gateway / Hub, but again these products will become available worldwide. Here is link to the Swedish links:
http://www.ikea.com/se/sv/catalog/categories/departments/living_room/36812/
http://www.ikea.com/se/sv/catalog/products/40337806/
http://www.ikea.com/se/sv/catalog/products/80338960/
http://www.ikea.com/se/sv/catalog/products/80338941/
http://www.ikea.com/se/sv/catalog/products/80349888/
Link are in Swedish for Ikea Sweden site, but the PDF manuals on each page are available in English and many more languages, however they don't say much other than mounting instructions.
http://www.ikea.com/ms/en_US/img/buying_guides/fy17/april/Home_Smart_lighting_Buying_guide_APRIL1.pdf
Reason why I think that this news being interested is Ikea's aggresive pricing might them the first to make two-way communication home automation really affordable for almost everyone while still following all the electrical safety and wireless communications regulations in all countries, as they are today well known to have very low prices yet good manufacturing quality items.
UPDATE 1: Sound as Ikea Trådfri Gateway software uses the Cypress WICED IoT platform SDK (formerly Broadcom WICED IoT platform before acquired by Cypress http://www.cypress.com/internet-things-iot ) and have choosen to base their implementation on OMA (Open Mobile Alliance) and Eclipse recommended IoT protocol standards of those three logical components; CoAP (coaps) and DTLS layers of the LwM2M (Lightweight machine-to-machine) security model for IoT device management and protocol stack, using IPSO (IP for Smart Objects) Smart Object Guidelines provide a common design pattern.  That is, looks as if the communication between Android/iOS app and the Gateway takes place via OMA Lightweight M2M (LwM2M) wrapped in CoAP with DTLS.
http://openmobilealliance.org/data-models-for-the-internet-of-things/
Update 2: Jaime Jiménez (from the company Ericsson) who is an active member of the IPSO Alliance’s working group and part of the team that published the IPSO Smart Object Guidelines, posted this great teardown of the Ikea Trådfri implementation:
http://jaimejim.github.io/tradfri/
For those who don’t know, LWM2M is a protocol built around CoAP and use for managing devices. So things like firmware upgrades, error reports, etc. Apart form the management interfaces, LWM2M also adds a very simple Object Model for managing those devices. IPSO expands that set of Objects so that you can have application information too (e.g. sensor readings, commands, etc). IPSO defines objects and resources that map to device properties.
Particular pay attention to the IPSO Light Control objects:
https://github.com/IPSO-Alliance/pub/blob/master/reg/xml/3311.xml
If you want to know more about the wealth of data models out there you can check the IoTSI Workshop as a reference.
https://www.iab.org/activities/workshops/iotsi/
LwM2M (Lightweight machine-to-machine) meanwhile is a system standard in the Open Mobile Alliance. It includes DTLS, CoAP, Block, Observe, SenML and Resource Directory and weaves them into a device-server interface along with an Objects structure based on IPSO Smart Object Guidelines.
https://connect2.io/open-mobile-alliance-lightweightm2m-oma-lwm2m/
https://www.eclipsecon.org/na2014/sites/default/files/slides/Eclipsecon NA14 - One protocol to rule them all- (1).pdf
http://openmobilealliance.org/constrained-application-protocol-coap-is-iots-modern-protocol/
IPSO provide common object model for interoperability of IoT Devices and Applications.
https://github.com/IPSO-Alliance/pub/blob/master/reg/README.md
https://github.com/IPSO-Alliance/pub/tree/master/reg/xml
https://github.com/IPSO-Alliance/pub/blob/master/reg/xml/3311.xml
https://github.com/IPSO-Alliance/pub

@Hedda Those are really affordable! Pretty much $17 USD for each item.
So, Ikea would need to publish an API that can be called for the ha-bridge to connect. We will need to keep looking into that.

From this link:
http://www.ikea.com/ms/sv_SE/customer-service/about-our-products/smart-lighting/index.html
Zigbee lights should also work with other bridges like Philips Hue, Osram Lightify or Vera Plus

Cool.  The more companies building smart home items the more it should drive down competition.  I do wish these were zwa e but I have a very plus so zigbee will be just fine for now.

While it is cool that Ikea have released a gateway/hub on their own I would personally much rather skip buying the Ikea Trådfri Gateway and instead buy and install an ZigBee USB-adapter in my mini-PC that I currently use for as my DIY home automation hub with Linux to control each Ikea Tråd Lightbulb and Lightpanel directly.
Problem with that simplter concept is that right now I don't know of which if any ZigBee USB-adapters and software libraries available that will work the ZLL (ZigBee Light Link) protocol which the Ikea Trådfri Lightbulbs and Lightpanels. Anyone here got some advice there?

There's a user on the hue developers forum that said it works with the old hub 1.0.
https://developers.meethue.com/content/philips-hue-and-ikea-trådfri

@emiliosic, Ikea Trådfri sensors and devices uses ZLL (ZigBee Light Link) protocol so will probably be made to work sooner or alter with hubs such as Samsung SmartThings Hub and Vera Plus. That is because those hubs are made by companies who made it the main concept for those hubs as commercial products to work with as many third-party devices as possible. That is, their goal is to make the hubs compatible with third-party sensors and devices. Samsung and Vera does officially support third-party devices.
But that is not the goal of the Philips Hue and Osram Lightify hubs. Thier goal is only for those hubs to work with their own devices, which means that they do not do compatibility testing with third-party ZigBee devices connected to their hubs. So while some third-party sensors and devices might work because they too use the ZLL (ZigBee Light Link) protocol, is not the goal of Philips or Osram to make third-party devices work stable with their hubs, as that is not in their interest. Philips and Osram want people to buy their own devices, so officially they only support their own devices connected to their hubs.

If the lights follow the standard there shouldn't be any development needed. Same goes for many Z-Wave devices which just work as generic.
http://www.zigbee.org/zigbee-for-developers/applicationstandards/zigbee-light-link/
I mentioned the lightify bridge because I noticed that HomeSeer seems to have decided to use a lightly bridge instead of developing their own:
http://www.homeseer.com/zigbee.html
But agree, if you have a Vera, SmartThings or Wink, it would probably work either at launch or soon enough.

I believe what the OP intended is, will the ha-bridge support the Tradfri Gateway. This is a specific hub and will need an API to talk to. This Gateway would be on the level of the vera, wink hub or smartthings hub.

@bwssytems That is exactly what I mean. Idea is that ha-bridge should be bridge to Ikea Trådfri Gateway. I did not mean that ha-bridge should communicate using ZigBee directly to the devices.
Amazon Echo => ha-bridge => Ikea Trådfri Gateway => Ikea Trådfri Lightbulbs/Lightpanels.
Google Home => ha-bridge => Ikea Trådfri Gateway => Ikea Trådfri Lightbulbs/Lightpanels.
Harmony Hub => ha-bridge => Ikea Trådfri Gateway => Ikea Trådfri Lightbulbs/Lightpanels.
Main problem with this is that we don't know if the Ikea Trådfri Gateway will offer a (documented) open API on the day-one of release. However I an interview with Ikea smart home development team from 6-months ago where they first leaked the news of plans to release Ikea Trådfri Gateway and in that article the Ikea developers specifically said that they do plan for the Ikea Trådfri Gateway to have an open API.
So if it an open API not available in the initial firmware pre-installed day-one of release, they might release an over-the-air update for the firmware sooner or later which will offer an open API. And even if they might not officially offer support to third-parties that uses that API once available, the Ikea developers of this product have at least stated they do want interoperability with third-party smart home solutions.
One possible option if the Ikea Trådfri Gateway does not offer an open API in the initial firmware pre-installed day-one of release is to hack the network communication between Ikea Trådfri Android/iOS application and the Ikea Trådfri Gateway, and then make library for ha-bridge software that can emulate an Ikea Trådfri Android/iOS app, similar to how ha-bridge software today emulate a Philips Hue Hub. That would of course be much more difficult and might not something that the developers of ha-bridge have any personal interest to dvelve into, but with the Ikea Trådfri series gateway and devices being so cheap and sold worldwide it will surley not be long before some other hacker hacks the communication between the Ikea Trådfri Android/iOS apps and the Ikea Trådfri Gateway and then publish that information or even code/library publicly for how to achieve exactly this. At it is several hackers usually have more insentive to hack stuff like this when a company does not offer an open API. Again, just look at the ha-bridge software and the Philips Hue Hub API.
Here are some more decriptive diagrams with paths to put these concepts in perspective of ha-bridge:
Amazon Echo => Hue API in ha-bridge core => API by Ikea Trådfri Gateway => Ikea Trådfri device to control
Google Home => Hue API in ha-bridge core => API by Ikea Trådfri Gateway => Ikea Trådfri device to control
Harmony Hub => Hue API in ha-bridge core => API by Ikea Trådfri Gateway => Ikea Trådfri device to control

The app is now downloadable on the google play store and apple app store , in case anyone wants to analyse it.
https://play.google.com/store/apps/details?id=com.ikea.tradfri.lighting
https://appsto.re/gb/NkWrhb.i
I have tried to start it in a virtual machine but it just complains that im not connected to wifi, then i tried to start it in arc welder, i came a little bit farther but then it just crashed.

Has any one tried to dump the network packets leaving the trådfri app? (i don't have access to a physical android phone)

@AnderssonPeter Tried to capture the packages from the app on my Android but tPacketCapture did not allow the Trrådfri app to work so I need another way to capture packages.. Also neither the iOS or Android app use any connection over HTTP what I could see when trying to capture req/resp.

@AnderssonPeter maybe try loading Ikea Trådfri app in the Android Emulator?
https://developer.android.com/studio/run/emulator.html
You of course need to have to have a physical Ikea Trådfri Gateway pre-installed.
But not sure if can enable WiFi in Android Emulator to local network though?
https://www.techwalla.com/articles/how-to-enable-wifi-on-an-android-emulator
If that works then should be able to use Wireshark to capture all IP packages
https://www.wireshark.org

@Hedda i don't have the official android emulator installed as that requires java and i don't want all those nagging java update popups.
I have tried with the visual studio android emulator, but none of the package capturing apps seem to work there sadly.
And running wire shark on the host did nothing..
I have also tried android x86, but there i ran into the wifi problem and when i tried a solution to add wifi or fake wifi the emulator got stuck in a reboot loop..

@AnderssonPeter  You can install Android x86 in a virtual machine on VirtualBox or KVM
https://www.howtogeek.com/164570/how-to-install-android-in-virtualbox/
http://www.osboxes.org/android-x86/
http://www.android-x86.org/

@Hedda i had it up and running on hyper-v and it worked until i managed to brick it when trying to fake the WIFI..

Think you'll find Tradfri uses ZHA not ZLL. when IKEA stated they were Zigbee compliant they were, just using 'the other' protocol. Most of the other hubs (like the hue hub 2.0) use ZLL though i think home hub 1 can use ZHA with an old firmware. All it really means is don't rush out and buy some bulbs and expect them to work with your hue hub 2.0. (yet)

From what I could see (proxy:ing the app traffic) it's using DTLS v1.2. I can see a "Hello" from the client and then an encryption request. After that all data is encrypted. If I understand it all correctly it's using a PSK. I.e. we'd need to extract that to be able to talk to the gateway. I'm in no way good at network communication so I might be completely wrong on this.

@stenehall you are correct it is using DTLS, some analysis has stared here: https://community.home-assistant.io/t/ikea-tradfri-gateway-zigbee/14788/8

Someone now looks to have 'cracked' the Ikea Trådfri Gateway - Turns out it communicates using standard CoAP encrypted using DTLS
https://bitsex.net/software/2017/coap-endpoints-on-ikea-tradfri/
https://bitsex.net/software/2017/ikea-tradfri-zigbee-lights/
This Norwegian guy as linked in two blog posts above have already figured out how monitor and send control commands to the Ikea Trådfri Gateway using standard CoAP to send and recieve commands to its end devices (using DTLS based encrypted communication to the Ikea Trådfri Gateway over network).
Ikea Trådfri Android application appears to be using multicast (<IP_ADDRESS>) to find the Ikea Trådfri Gateway, and then communicates using encrypted CoAP (coaps). Also, it does not look like the Trådfri Gateway attempts to talk to the Internet (as the device looks to have has no outgoing connections). The default Ikea Trådfri Android app is fairly basic, where it currently only let you create schedules for turning on and off, and you can control lights and create zones, and control zones.
Intestingly is that CoAP (Constrained Application Protocol) is the protocol which the OCF (Open Connectivity Foundation) backed by most industry gians is promoting to become the standard for IoT. As reference in IoTivity and AllJoyn projects which guidelines one can suspect that Ikea have followed when they choose to go with standard CoAP in the Ikea Trådfri Gateway and its apps.

I'm that norwegian guy. I was not aware of the standard, and I'm happy that you brought it to my attention.
I was not aware of ha-bridge either, but it seems interesting. So far I've been a bit stumped by that zero of the COAP libraries for python supports dTLS, and I don't want to go the java route... I got a couple more bulbs today, and confirm that they add the same place as the previous.
If I can be of any assistance, I'm happy to. If wanted I can provide a linux box with access to the gateway for anyone wanting to play.
The key that is used for communication is printed beneath the gateway, so there's no need for any cracking; it's perfectly open, albeit using a not-yet-so-common protocol.

Hi guys,
You can easily talk to the gateway if you build the "dtls" branch of https://github.com/obgm/libcoap
And then enter examples and do stuff like:

echo '{ "3311" : [{ "5851" : 255 }] }' | ./coap-client -u "Client_identity" -k "YOUR_KEY" -v 10 -m put "coaps://192.169..0.3:5684/15001/65538" -f -

You can set the dimmer to something between 0-255.
You can also query all the available endpoints:

./coap-client -u "Client_identity" -k "YOUR_KEY" -v 10 -m get "coaps://<IP_ADDRESS>:5684/.well-known/core"

@vidarlo Sound as Ikea have choosen to base their implementation on Eclipse recommended standard of those three logical components; CoAP, and DTLS layers of the LwM2M protocol stack.
https://iot.eclipse.org/standards/
https://eclipse.org/community/eclipse_newsletter/2014/february/article2.php
The Wakaama project covers the LWM2M Protocol, CoAP, and DTLS layers of the LwM2M protocol stack for all three logical components. Wakaama is not a library but files to be built with an application. The Eclipse Wakaama project provides a C portable framework for building LWM2M clients and/or servers. The source code of Wakaama is available from the project webpage. It is written in C and designed to be portable on POSIX compliant systems.
http://www.eclipse.org/wakaama/
The Eclipse Leshan project provides a Java implementation of LwM2M, allowing to build LwM2M servers and clients. The source code of Leshan is available from the project webpage.
http://www.eclipse.org/leshan/
Another option from Eclipse is Californium CoAP client programmed in Java which also supports DTLS
https://eclipse.org/californium/
https://github.com/cetic/6lbr/wiki/Example-:-Dtls-Coap-Server
https://people.inf.ethz.ch/mkovatsc/resources/californium/cf-dtls-thesis.pdf

OMA LightweightM2M (LWM2M) standard:
http://openmobilealliance.org/iot/
http://openmobilealliance.org/iot/lightweight-m2m-lwm2m/
http://www.openmobilealliance.org/wp/Overviews/lightweightm2m_overview.html
http://www.openmobilealliance.org/wp/OMNA/LwM2M/LwM2MRegistry.html
http://www.openmobilealliance.org/tech/profiles/
https://github.com/OpenMobileAlliance/OMA_LwM2M_for_Developers/wiki
http://devtoolkit.openmobilealliance.org/OEditor/Legal?back=Default
http://www.openmobilealliance.org/wp/comments.html
https://github.com/OpenMobileAlliance/OMA_LwM2M_for_Developers/issues
http://openmobilealliance.hs-sites.com/keep_updated

@arturo182 tinydtls-coap is another attempt to integrate the libcoap and tinydtls client-server
https://github.com/thecodemaiden/tinydtls-coap
You are then missing implementation of the LWM2M standard an top of CoAP.

@Hedda, yes I have tried that one, it has the psk hardcoded and even when I changed that I still couldn't get a DTLS handshake, the dtls branch of libcoap works very well.

Bought my Trådfri starter kit today (in Germany where it was launched shortly ago) and have been following this thread attentively, nice insights so far :)
The advice from @arturo182 was worth gold, but building libcoap was a little troublesome, so I summarized the installation in the following, might save others some time when experimenting:
#!/bin/bash
git clone https://github.com/obgm/libcoap.git
cd libcoap
git checkout origin/dtls
git checkout -b dtls
git submodule update --init ext/tinydtls
cd ext/tinydtls
autoreconf
./configure
cd ../../
./autogen.sh
./configure --disable-shared
make

I would recommend doing:
export GATEWAY_IP="192.168.XXX.YYY"
export GATEWAY_KEY="<gateway key from underside label>"

Then you can get available endpoints (from @arturo182):
./coap-client -u "Client_identity" -k "${GATEWAY_KEY}" -v 10 -m get "coaps://${GATEWAY_IP}:5684/.well-known/core"

or use dimming:
echo '{ "3311" : [{ "5851" : 255 }] }' | ./coap-client -u "Client_identity" -k "${GATEWAY_KEY}" -v 10 -m put "coaps://${GATEWAY_IP}:5684/15001/65538" -f -

this dims one of my two bulbs (the one I set up first, they are in different groups).
3311 is an ext-label for dimming, 5851 stands for a dimmer (0-100%), but the value after that seems to be 0-255.
UDP 5684 is the port used by CoAP. 15001 seems to be the number of the endpoint used by the bridge and 65538 is the identifier for an actual bulb. My bulbs had the numbers 65537 and 65538 and there's something with the number 65536, this may be the remote that was used to pair the bulbs.
Device number seem to start at 65536.
Most valuable resource so far was: http://www.openmobilealliance.org/wp/OMNA/LwM2M/LwM2MRegistry.html

Bought my Trådfri starter kit today (in Germany where it was launched shortly ago) and have been following this thread attentively, nice insights so far :)
The advice from @arturo182 was worth gold, but building libcoap was a little troublesome, so I summarized the installation in the following, might save others some time when experimenting:
#!/bin/bash
git clone https://github.com/obgm/libcoap.git
cd libcoap
git checkout origin/dtls
git checkout -b dtls
git submodule update --init ext/tinydtls
cd ext/tinydtls
autoreconf
./configure
cd ../../
./autogen.sh
./configure --disable-shared
make

I would recommend doing:
export GATEWAY_IP="192.168.XXX.YYY"
export GATEWAY_KEY="<gateway key from underside label>"

Then you can get available endpoints (from @arturo182):
./coap-client -u "Client_identity" -k "${GATEWAY_KEY}" -v 10 -m get "coaps://${GATEWAY_IP}:5684/.well-known/core"

or use dimming:
echo '{ "3311" : [{ "5851" : 255 }] }' | ./coap-client -u "Client_identity" -k "${GATEWAY_KEY}" -v 10 -m put "coaps://${GATEWAY_IP}:5684/15001/65538" -f -

this dims one of my two bulbs (they are in different groups).
3311 is an ext-label for dimming, 5851 stands for a dimmer (0-100%), but the value after that seems to be 0-255.
UDP 5684 is the port used by CoAP. 15001 seems to be the number of the endpoint used by the bridge and 65538 is the identifier for an actual bulb. My bulbs had the numbers 65537 and 65538 and there's something with the number 65536, this may be the remote that was used to pair the bulbs.
Device number seem to start at 65536.
Most valuable resource so far was: http://www.openmobilealliance.org/wp/OMNA/LwM2M/LwM2MRegistry.html

OK, I now played a bit around with the Anjay LwM2M library and I would recommend it to anyone doing LwM2M, the documentation is just awesome.
Link: https://github.com/AVSystem/Anjay
Compilation is easy and is explained in the README.
Here's what I tried so far:
As a preparation I would suggest again:
export GATEWAY_IP="192.168.XXX.YYY"
export GATEWAY_KEY="<gateway key from underside label>"

Then, use the demo binary that was created during compilation.
Connecting to a coaps-DTLS server requires quite some parameters,
Identity and Key need to be supplied "hexlified", which xxd -p poes.
./output/bin/demo \
    --server-uri "coaps://${GATEWAY_IP}:5684" \
    --security-mode psk \
    --identity `echo -n "Client_identity" | xxd -p` \
    --key `echo -n $GATEWAY_KEY | xxd -p`

This prints a lot of successfully registered object and then manufacturer: 0023C7; serial number: 000001, that doesn't look bad, but I haven't gotten any further yet, but the above command should server as a good starting point.

FYI; pautomate from the Home Assistant community put together a simple Python implementation for his Raspberry Pi.
https://community.home-assistant.io/t/ikea-tradfri-gateway-zigbee-very-basic-working-implementation/14788/19
"Taking no credit of all the hard work others have put in and just put my code here if you want to play around":
First installed the libcoap-library as such:
apt-get install libtool
git clone --recursive https://github.com/obgm/libcoap.git
cd libcoap
git checkout dtls
git submodule update --init --recursive
./autogen.sh
./configure --disable-documentation --disable-shared
make
sudo make install
pautomate put together a very simple Python-script that execute by running shell_commands:
https://github.com/ggravlingen/home-assistant/blob/master/extraconfig/python_code/ikea.py
You then run the code by invoking:
python ikea.py "65537"  "100" "Yellow"
Or, from within Home assistant as a shell command:
ikealight_off: '/usr/bin/python3 /home/homeassistant/.homeassistant/extraconfig/python_code/ikea.py "65537"  "0" "Yellow"'

@r41d  FYI, Ikea is a customer of AVSystem who developed the LwM2M library Anjay (which is dual-licensed, bot as free and open source software and under a commercial license).
https://www.avsystem.com/news/avsystem-anjay-supports-lwm2m1/
https://www.avsystem.com/products/anjay/
While they don't say so outright one could suspect that Ikea uses commercially licensed Anjay and/or AVSystem Coiote for the Ikea Trådfri Gateway and its apps.
AVSystem Coiote it is a complete end-to-end solution for empowering IoT applications, which I understand relies on Anjay too.
https://www.avsystem.com/products/coiote/
https://avsystem.github.io/Anjay-doc/Commercial_support.html

I found an intersting class in the Android App, after decompliling with apktool, it is located in com/ikea/tradfri/lighting/ipso/IPSOObjects.java.
Here's an upload: http://sprunge.us/CCQF
This lists a lot of constants that one can see when talking to the bridge (with this (https://bitsex.net/software/2017/coap-endpoints-on-ikea-tradfri/) in mind).

I just wrote a minimal web interface for controlling the brightness of Trådfri bulbs.
It's absolutely dirty code, but I just wanted an working example to start with for now.
You need to install libcoap with dtls like I described here: https://github.com/bwssytems/ha-bridge/issues/570#issuecomment-292036357
You also need to have Flask installed.
Then you can download the server https://gist.github.com/r41d/65be2c7a111ac6c32f24d762ba38612c
Place it one the same level in the file system as the libcoap folder and heed the instructions in the file.
The interface is ugly as hell, but it's a first step.
Have fun.
I also posted the instructions here: https://community.home-assistant.io/t/ikea-tradfri-gateway-zigbee-very-basic-working-implementation/14788/20

If also create a simple python script for reading the status of the lightbulbs that are paired with the hub. Still working on a framework that can get the status and put new state(s) to the lightbulbs.
Code is not up to my spec, but will post it soon.
script for reading lightbulb state: https://gist.github.com/hvanderlaan/3d8e11869f86ba94d9d6df1c815af3aa

I spent some time writing a small homebridge plugin for this. https://github.com/stenehall/homebridge-ikea it works with some kinks.

@stenehall nice but unfortutante that ha-bridge can not yet connect to Homebridge as a HomeKit bridge.
That is however a frequently asked feature request for ha-bridge:
https://github.com/bwssytems/ha-bridge/wiki/HA-Bridge-FAQs#Hb_Hk
https://github.com/bwssytems/ha-bridge/issues/366
https://github.com/bwssytems/ha-bridge/issues/159
If it could then ha-bridge could indirectly take advantage of any endpoints that Homebridge supports too.
Amazon Echo => ha-bridge => Homebridge => Ikea Trådfri Gateway => Ikea Trådfri devices or HomeKit devices
Google Home => ha-bridge => Homebridge => Ikea Trådfri Gateway => Ikea Trådfri devices or HomeKit devices
Harmony Hub => ha-bridge => Homebridge => Ikea Trådfri Gateway => Ikea Trådfri devices or HomeKit devices
Even nicer would be two-way bridge between ha-bridge & Homebridge so both could use each others endpoints.
I know it's off-topic but full two-way compatibility between ha-bridge & Homebridge would just be awesome!
Google Home => ha-bridge => Homebridge => All the Homebridge plugins for endpoints including HomeKit
HomeKit => Homebridge => ha-bridge = All the ha-bridge plugins for endpoints

I have been working on reverse engineering the HAP so that the ha-bridge  can talk to homebridge or any other HAP device. The only thing is being the client side, there are no libraries and I am working with the server side. Currently, I am not getting consistent handshake results with the encryption.

@Hedda I'm actually not using ha-bridge, only using homebridge atm. But since I'd followed this issue and commented in it I figured I'd provide what I had in terms of code :)

Learn more about IPSO objects at:
https://github.com/IPSO-Alliance/pub/blob/master/reg/README.md

Matthew Garrett (formerly of Broadcom) have posted an analysis of the security of the gateway here:
https://mjg59.dreamwidth.org/47803.html
He seela sure the gateway is based on the Cypress WICED IoT platform:
http://www.cypress.com/internet-things-iot

I don't know if this has been mentioned in the long thread. The numbers you see are LWM2M/IPSO objects.
http://ipso-alliance.github.io/pub/
I wrote a blog post on that here, I am still working with the GW itself.
http://jaimejim.github.io/tradfri/

Was playing with the Tradfri lights today. Had some weird behaviour though. I was able to query the lights' state no problem, and got the expected response back but when trying to send a put request to control the lights I just got back a 4.00 response. I had two bulbs and a remote connected to the hub.

Thanks for all the effort! I've played with IKEA app and some of the devices, I can help reverse engineer some of the values in API.

@sandyjmacdonald if you are using groups (multiple lights) you have to address the group. This is not coaps://:5684/15001 but coaps://:5684/15004
you could take a look at my python code, this will show information of the lightbulbs and groups. It also can control lightbulbs and groups. You can power on and power off lights and groups, dim lights and groups and change color temperature of the lights only. Color temperature of groups are done by presets. I've not yet incorporate these. This will follow later this week when I have some more time.
Feel free to clone or fork my repo: https://github.com/hvanderlaan/ikea-smartlight

I've created simple (and kind of dumb) tool for Tradfri output to be more readable:
https://github.com/matemaciek/tradslator/blob/master/tradslator.py
I've used names from http://sprunge.us/CCQF and added some new ones basing on experiments with my setup.
Examples of its results:
$ ./libcoap/examples/coap-client -u "Client_identity" -k "${GATEWAY_KEY}" "coaps://${GATEWAY_IP}:5684/.well-known/core" | tr "," "\n" | python ./tradslator/tradslator.py
v:1 t:CON c:GET i:ec26 {} [ ]
decrypt_verify(): found 24 bytes cleartext
decrypt_verify(): found 935 bytes cleartext
<//{DEVICES}/65541>;ct=0;obs
<//{DEVICES}/65543>;ct=0;obs
<//{DEVICES}/65536>;ct=0;obs
<//{DEVICES}/65537>;ct=0;obs
<//{DEVICES}/65540>;ct=0;obs
<//{DEVICES}/65542>;ct=0;obs
<//{DEVICES}/65539>;ct=0;obs
<//{DEVICES}/65538>;ct=0;obs
<//{GROUPS}/131602>;ct=0;obs
<//{GROUPS}/153539>;ct=0;obs
<//{SCENE}/153539/203088>;ct=0;obs
<//{SCENE}/153539/203292>;ct=0;obs
<//{SCENE}/153539/209306>;ct=0;obs
<//{SCENE}/131602>;ct=0;obs
<//{SCENE}/153539>;ct=0;obs
<//{SCENE}/131602/205574>;ct=0;obs
<//{SCENE}/131602/209107>;ct=0;obs
<//{SCENE}/131602/217767>;ct=0;obs
<//{SCENE}/153539/228632>;ct=0;obs
<//{SCENE}/153539/199834>;ct=0;obs
<//{SCENE}/153539/217483>;ct=0;obs
<//{DEVICES}>;ct=0;obs
<//{DEVICES}/reset>;ct=0
<//status>;ct=0;obs
<//{SCENE}>;ct=0;obs
<//{GROUPS}>;ct=0;obs
<//{GROUPS}/add>;ct=0
<//{GROUPS}/remove>;ct=0
<//{NOTIFICATIONS}>;ct=0;obs
<//{GATEWAY}/{GATEWAY_DETAILS}>;ct=0;obs
<//{GATEWAY}/{UPDATE_FIRMWARE}>;ct=0
<//{GATEWAY}/{REBOOT}>;ct=0
<//{GATEWAY}/{RESET}>;ct=0
<//{GATEWAY}/{AUTH_PATH}>;ct=0
<//{GATEWAY}/{SESSION_ID}>;ct=0
<//{SCHEDULES}>;ct=0;obs
<//{SCHEDULES}/270213>;ct=0;obs

$ ./libcoap/examples/coap-client -u "Client_identity" -k "${GATEWAY_KEY}" "coaps://${GATEWAY_IP}:5684/15010/270213" | tr "," "\n" | python ./tradslator/tradslator.py
v:1 t:CON c:GET i:9d64 {} [ ]
decrypt_verify(): found 24 bytes cleartext
decrypt_verify(): found 169 bytes cleartext
{"{REPEAT_DAYS}":31
"{CREATED_AT}":1491151882
"{INSTANCE_ID}":270213
"{ONOFF}":1
"{SMART_TASK_TYPE}":4
"{START_ACTION}":{"{ONOFF}":1
"{LIGHT_SETTING}":[{"{INSTANCE_ID}":65538
"{DIMMER}":254
"{TRANSITION_TIME}":18000}]}
"{TRIGGER_TIME_INTERVAL}":[{"{START_TIME_HR}":3
"{START_TIME_MN}":30}]}

$ ./libcoap/examples/coap-client -u "Client_identity" -k "${GATEWAY_KEY}" "coaps://${GATEWAY_IP}:5684/15001/65540" | tr "," "\n" | python ./tradslator/tradslator.py
v:1 t:CON c:GET i:b7f7 {} [ ]
decrypt_verify(): found 24 bytes cleartext
decrypt_verify(): found 313 bytes cleartext
{"{NAME}":"Kuchnia"
"{CREATED_AT}":1491151779
"{LAST_SEEN}":1491860913
"{INSTANCE_ID}":65540
"{OTA_UPDATE_STATE}":0
"3":{"0":"IKEA of Sweden"
"1":"TRADFRI bulb E27 WS opal 980lm"
"2":""
"3":"<IP_ADDRESS>-<IP_ADDRESS>"
"6":1}
"{TYPE}":2
"{REACHABILITY_STATE}":1
"{LIGHT}":[{"{ONOFF}":1
"{DIMMER}":187
"5707":0
"5708":0
"{COLOR_X}":24930
"{COLOR_Y}":24694
"5711":0
"{COLOR}":"f5faf6"
"{INSTANCE_ID}":0}]}

It seems like decoder would be a nice addition, maybe next time.

@matemaciek nice, great stuff.
I have been experimenting with libcoap and decided to fork it for better Tradfri support: https://github.com/r41d/libcoap-tradfri
The repo contais a build script build_libcoap_with_tinydtls.sh which does all the work.
The new coap-tradfri example is just a variant of coap-client, but makes the invocation much easier.
It currently defaults to 15001 as a path prefix, but that be changed very easy in the source.
coap-tradfri 192.168.X.Y -b 65537 -k abcdefghij
coap-tradfri 192.168.X.Y -b 65537 -k abcdefghij -m put -e '{"3311":[{"5851":0}]}'

I also looked at other CoAP libraries that support DTLS (both also written in C):

LibNyoci claims to have experimental DTLS support, but the linked issue doesn't exists. Not sure if I want to grapple with it.
FreeCoAP looks like it has good source code quality and is also very small and lucid. But the DTLS examples there onyl use certificates, I wasn't able to figure out how to do PSK DTLS with it.

The last DTLS CoAP lib is the Eclipse Californium framework, which is also used by the official Android App. Even though it's Java, it seems like the best shot at the time.

Home Assistant developers ( ggravlingen /  pautomate, balloob & turbokongen ) looks to be making great progress with their collaberation on a Python class to communicate with the Ikea Trådfri Gateway, and also extracted that code from the platform into a standalone Python library
https://github.com/ggravlingen/python-openikeatradfri
More discussion on that in the Home Assistant Community forum
https://community.home-assistant.io/t/ikea-tradfri-gateway-zigbee-very-basic-working-implementation/14788/

I found some small quarckiness, in the gateway,

you do not need / in the coaps url coaps://<gateway>:5684/1500165537 wil also work perfect.
when you power off a group with the mobile app, the group power state will retain the last status. { "5850": 1 } but the lightbulbs in the group are off { "5850": 0 }.

The first could be a security issue, or possibly a way to get more information from the gateway because /15002, /15009, /15011, /15012, /15013 will give a 4.04 not found. I will investigate.
 Now is the only thing where do we need to address this bug.

Thanks so much @hvanderlaan!! That sorted the issue. You'll have seen our tweet using your code! :-) Do you know if there's a way to address bulbs individually when they're within a group?

@sandyjmacdonald Yes, you could change the state, dim or change the color individual from a group by addressing the lightbulb directly (/15001/)

Another developer making progress is @hardillb :
https://www.hardill.me.uk/wordpress/2017/04/06/fist-pass-tradfri-mqtt-bridge/
https://github.com/hardillb/TRADFRI2MQTT

The Ikea tradfri uses the iCoAP libs, this lib supports http/https proxy. Maybe in the future this will/could be enabled. When this is done the lib-coap / coap-client is not needed any more because you could talk to the http/https proxy.
https://github.com/stuffrabbit/iCoAP
If they enable this function the python module requests of urllib should be sufficent to talk with the hub.

https://www.hardill.me.uk/wordpress/2017/04/06/fist-pass-tradfri-mqtt-bridge/
A Java based mqtt bridge.  Useful for a Openhab Add on writer?

Did anyone actually try to reach out to IKEA and ask if they plan a public API documentation (similar to Philips hue)? It would make stuff much easier than having to guess how it works through reverse engineering...

Yes: https://twitter.com/hardillb/status/848644863885271040

I also had contact with Ikea, they say that in october they will add Apple's Home Kit Support. So there will be an api for this in the future. Also iCoAP can use a proxy but for now it is not enabled.

I also used Ikea's contact-form a couple of days ago to ask if Ikea (Sweden) have any news to shere about their plans for an open/documentred API, but I have not recieved any reply to that as of yet.
Regardless I would advice everyone to write to Ikea and ask them to open up the Trådfri Gateway API.
http://www.ikea.com/ms/en_US/customer_service/contact_us/email_request.html
http://www.ikea.com/gb/en/customer-service/contact-us/email/
http://www.ikea.com/ms/sv_SE/kundservice/kontakta_oss/maila_oss.html
https://customersupport-ikea-nl-nl.custhelp.com/app/ask/
http://www.ikea.com/ms/en_HK/customer_service/contact_us/contact.html
Would be good if Ikea get more feedback so they understand that not everyone use Apple HomeKit.

Would be good if Ikea get more feedback so they understand that not everyone use Apple HomeKit.

I fully agree - I'll try to do my part to make IKEA aware of the importance of open APIs.

I'm not sure it was in this thread, but I read that Ikea already confirmed
they are going to release an open API...
2017-04-12 12:21 GMT+02:00 Kai Kreuzer<EMAIL_ADDRESS>
Would be good if Ikea get more feedback so they understand that not
everyone use Apple HomeKit.
I fully agree - I'll try to do my part to make IKEA aware of the
importance of open APIs.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/bwssytems/ha-bridge/issues/570#issuecomment-293535628,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AZKlU6E4AU_GexRfXDCz71uAXEgRjvhIks5rvKWqgaJpZM4MqK37
.

@hvanderlaan
I'm having the same problems as @sandyjmacdonald, I can read values through coap-client, but I can't seem to control my lights; I only get 4.00 as a response and neither the 5850 nor the 5851 values change.
I've tried controlling both the individual lights and the group, but I can't seem to get it to work. I've also tried the Python class with no success.
The commands I've tried for controlling the lights are:
echo '{"3311":[{"5851":254}]}’ | coap-client -u "Client_identity" -k $KEY -m put "coaps://$IP:5684/15001/65537" -f -
and
echo '{"3311":[{"5850":1}]}’ | coap-client -u "Client_identity" -k $KEY -m put "coaps://$IP:5684/15001/65537" -f -
And for controlling the group:
echo '{"5851":254}’ | coap-client -u "Client_identity" -k $KEY -m put "coaps://$IP:5684/15004/166682" -f -
and
echo '{"5850":1}’ | coap-client -u "Client_identity" -k $KEY -m put "coaps://$IP:5684/15004/166682" -f -
4.00 as a response for all of the above. I'm starting to think I've got a typo somewhere in there...

Try it the following way, @andersmilton:
coap-client -m put -u "Client_identity" -k "1a2b3c4d5e6f7g8h" -e '{ "3311": [{ "5850": 1 }] }' "coaps://<IP_ADDRESS>:5684/15001/65537"

OMG, works like a charm. Thanks @sandyjmacdonald!

:-) Took me a while to suss that too. Not sure why the echo and pipe isn't working?

Perhaps a character encoding problem or something? What platform are you on? I'm running it from a macOS terminal window through ssh to another Mac, so I guess there's plenty of room for translation errors...

I'm SSHing into a Pi Zero W from a macOS terminal window.

Looks like macOS could be the problem. Oh, well. As long as it works :)

This is correct, it's an issues with macOS, on a ubuntu box i can use the echo way but the payload argument of coap-client is a very nice solution to use

My default OS X terminal encoding seems to be UTF-8. You can find out your encoding with the command locale. And I found this quite interesting:
"Now that we have checked the local terminal settings, we should do the same for hosts where we ssh into. Luckily, ssh can forward our locales settings, just append “SendEnv LANG LC_ALL” to ~/.ssh/config and check that your locale is also available on the remote host. Voila, you have a properly working terminal with defined locales." (found at https://benjamin-schweizer.de/unix-terminals-surviving-the-encoding-hell.html)

On Apr 12, 2017, at 12:45, Sandy Macdonald<EMAIL_ADDRESS>wrote:

coap-client -m put -u "Client_identity" -k "1a2b3c4d5e6f7g8h" -e '{ "3311": [{ "5850": 1 }] }' "coaps://<IP_ADDRESS>:5684/15001/65537”

Actually, you can leave off that :5684, that is implicit in the coaps:
Grüße, Carsten

I've put together a tutorial about how to set all this up on a Pi, using @hvanderlaan's Python code, and showing how to use one of our (Pimoroni) Enviro pHAT boards as a light-responsive trigger, for those that might be interested.
https://learn.pimoroni.com/tutorial/sandyj/controlling-ikea-tradfri-lights-from-your-pi

You can now also omit the -u for coap-client:
coap-client -m put -k "1a2b3c4d5e6f7g8h" -e '{ "3311": [{ "5850": 1 }] }' "coaps://<IP_ADDRESS>/15001/65537"

It doesn't seem to let me omit the -u, but I found an easy way of changing the color of the light by using the RGB hex values (f5faf6 , f1e0b5  or efd275) like so:
coap-client -m put -u "Client_identity" -k "$KEY" -e '{ "3311": [{ "5706": "f5faf6" }] }' "coaps://$IP/15001/65537"
I don't think I've seen that posted here before. I apologise if I'm mistaken :)

I just implemented Color Changing (warm/normal/cold) for bulbs in the MQTT bridge from @hardillb:
https://github.com/r41d/TRADFRI2MQTT/tree/features
Awesome foundation by the way :+1:
If my pull requests get accepted, this should be the most complete open source Tradfri tool yet =)

Some findings regarding /15011 a.k.a the gateway (mostly based on looking at the code of the Android app):

a POST on /15011/9031 performs a soft reset
a PUT on /15011/9034 is used for performing a firmware update, but it looks like this is not implemented yet (firmware payload is empty)
a PUT on /15011/15012 can be used to perform many different operations (check OTA, commissioning, update time…) on the gateway, e.g update the current time by PUTting a timestamp with key 9060, etc.
/15011/9033 is TBD
/15011/9063 (POST) appears to be used for re-keying the gateway, as the code seems to involve the generation and transfer of a new PSK

For those on linux based systems have a look at this...
[(https://www.hardill.me.uk/wordpress/2017/04/06/fist-pass-tradfri-mqtt-bridge/)]
It seems to rely on a coap library (hence the linux thing, unless I am mistaken) and creates a mqtt to tradfri bridge.  Obviously it would be better if we had a binding that did it natively (and was cross platform)

@Justblair it's all pure java so should run (nearly) anywhere

Some inside news from IKEA, nothing official, it can be changed, don't get attached to them:

they're working on open API,
new products are to appear in October,
even more products in April next year.

I can't say, what products they're going to introduce, but  let's just say, that I think that RGB bulb and controllable plug sockets would be a nice addition to the system (-:

Oh I see.  I misunderstood the coap library thing.

On Thu, 13 Apr 2017, Maciej Sokołowski wrote:

I can't say, what products they're going to introduce, but let's just say,
that I think that RGB bulb and controllable plug sockets would be a nice
addition to the system (-:

Do you have any information regarding the number of devices? I've seen a
maximum of ten devices connected - and this seems low in some cases.

There is a limit of ten groups per gateway and ten devices per group, so limitation is not that painful.

There is a limit of ten groups per gateway and ten devices per group, so
limitation is not that painful.

Ok, so we can connect 100 devices theoretically?
Do you happen to know if the gateway supports observation, as per RFC7641?
That would be a quite nice addition, as Ikea makes motion sensors and
dimmers for Trådfri.

Observations work, but observing a dimmer does not give the level changes, I've not worked out what it is yet, but I suspect it might be the battery level

RGB bulbs from IKEA would be awesome, there is also already a hint in the code: the COLOR_X and COLOR_Y constants in the Tradfri Android App.
It's possible to express color with X and Y coordinates with the CIE 1931 color space: https://en.wikipedia.org/wiki/CIE_1931_color_space#CIE_xy_chromaticity_diagram_and_the_CIE_xyY_color_space
The following values from the CoAP traffic were extracted:
	public static final String COLOR_COLD = "f5faf6";
	public static final String COLOR_COLD_X = "24930";
	public static final String COLOR_COLD_Y = "24694";
	public static final String COLOR_NORMAL = "f1e0b5";
	public static final String COLOR_NORMAL_X = "30140";
	public static final String COLOR_NORMAL_Y = "26909";
	public static final String COLOR_WARM = "efd275";
	public static final String COLOR_WARM_X = "33135";
	public static final String COLOR_WARM_Y = "27211";

Changing color temperatures doesn't need any of the X or Y values, but if we look at this picture https://upload.wikimedia.org/wikipedia/commons/3/3b/CIE1931xy_blank.svg
I'm not sure how the scaling works, we have 0-1 in the linked picture but 5 digit integers in the CoAP traffic, 65535 might be the maximum there.
Cold = 24930,24694. Both values are almost similar. Ergo: near to white.
Normal = 30140,26909. Greater values in both X and Y (though much more increase in X), it's probably somewhere in that yellow range.
Warm = 33135,27211. Again, much more increase in X than in Y, it's probably in the orange area.
Does that sound reasonable?

Yeah @r41d , that seems like a good approximation.
I must find some time to try some colors in the middle of Tradfri bulbs range. Maybe we can set more than three colors via CoAP? I dissected one of my bulbs, it has two colors of emitters, warm and cold, and, of course, that makes three available color temperatures. But I'm wondering whether we can set them to different brightness to achieve the whole spectrum between cold and warm white.

You do indeed have full control over the colour temperature of the bulbs, and not just the three temps! Here's an example that ramps back and forth through the range:
import os
import time
import math

coap = '/usr/local/bin/coap-client'
securityid = 'xxxxxxxxxxxxxxxx'
hubip = '192.168.x.x'
lightbulbid = '65538'
tradfriHub = 'coaps://{}:5684/15001/{}'.format(hubip, lightbulbid)

while True:
    scale_factor = (1 + math.sin(time.time() * 2)) / 2
    payload = '{ "3311" : [{ "5709" : %s, "5710": %s }] }' % (25000 + int((8000 * scale_factor)), 25000 + int((2000 * (1 - scale_factor))))
    api = '{} -m put -u "Client_identity" -k "{}" -e \'{}\' "{}"' .format(coap, securityid, payload, tradfriHub)
    result = os.popen(api)
    time.sleep(0.01)

For the integration with Home Assistant we made the following Python lib:   https://github.com/ggravlingen/pytradfri/ . The Python abstraction maps the API 1:1
These are the constants that we have been able to figure out thus far

@balloob You can also have a look at http://sprunge.us/CCQF
These are constants that I extracted from the Android Tradfri App, class com/ikea/tradfri/lighting/ipso/IPSOObjects.java
@matemaciek also used them in his tradslator I think.

Hmm... everything seems to be broken for me. I wonder if IKEA pushed an update that somehow broke things. Not even querying the API with a get request from coap-client is working. :-(

Hmm... everything seems to be broken for me. I wonder if IKEA pushed an
update that somehow broke things. Not even querying the API with a get
request from coap-client is working. :-(

Which firmware version do you have? My GW has 1.1.0015 and works perfectly
fine.

FYI, Pimporoni posted step-by-step instructions for how pne could implement @hvanderlaan & @sandyjmacdonald Phyton code here:
https://learn.pimoroni.com/tutorial/sandyj/controlling-ikea-tradfri-lights-from-your-pi
They make it very easy to follow if someone who is not developer just want to try it out to test the functions.

Ops, didn not mean to close this, reopening.

Have fixed my problems, @vidarlo. It seems to be that one of the bulbs got stuck while updating and that was causing no requests to get through. Had to reset everything, pair everything again, and it all works as expected now.

All these things works great if you have lights and groups set up, i'm trying to read the value directly off a remote/dimmer, but i'm suspecting that it doesn't work like that...
Does the hub know if a remote is clicked on or does that communication happen directly to the lightsource then the lightsource broadcasts it's state to the hub so that the app can read it?
Or is the only way to do that to build a zigbee device?

@r41d I figured out that the colorX and colorY (5709 and 5710) range up to 10000, because in the CIE color space the values range between 0-1, and here they're probably just scaled up.
With this in mind, I created a homebridge plugin that has full support for the Ikea gateway and light bulbs, including dimming and accurate color changing, natively from the Home app: szokeptr/homebridge-tradfri-plugin

@szokeptr Hmm, did you look at my computations in my previous post and the correlation with the wikipedia picture? From that it looks much more likely that the maximum is 2^16=65535.
But I can't say anything definitive about this as I didn't do any testing like you or @sandyjmacdonald did, should make up leeway.
I think the most interesting part now, is which ranges are actually compatible to control the temperature.

@r41d yes, it looks like you're right, however this way my method is not that accurate. I am actually converting the hue and saturation values from the Home app to RGB then to CIE XY values. The function I am using is outputting values in the range of 0-1, and when I scaled that up to 0-100000 the lights are following the color changes almost dead accurate. I changed the scaling factor to 65535 and this way the colors are all in a much colder zone. I'll do some more testing on this.

@possan As described here: https://github.com/ggravlingen/pytradfri/issues/2:
You can indeed listen (subscribe) to changes on a bulb with -s
coap-client -u "Client_identity" -k {KEY} -s "60" "coaps://{IP}:5684/15001/65537"
This will leave the coap-client running and pop out new messages if there are changes, e.g. by clicking on any button on the connected remote.

Jaime Jiménez ( @jaimejim  from the company Ericsson) who is apparently an active member of the IPSO Alliance’s working group, and especially worked in the team that published the IPSO Smart Object Guidelines, posted this great teardown of the Ikea Trådfri implementation:
http://jaimejim.github.io/tradfri/
He also shared a couple of links to related articles he posted earlier:
http://jaimejim.github.io/lwm2m-yang/
http://jaimejim.github.io/coap-functionality-lwm2m/
Interestingly enough Michael Koster (who is the main author of these IPSO Objects) posted a comment to Jaime's post staying: "Great move on IKEA's part here. Too bad we hadn't published the IPSO lighting objects yet. Maybe we can talk them into registering the new identifiers with OMNA.".

@mjkoster is by the way the original author of iottoolkit - an IoT Toolkit with SmartObject API reference implementation and framework written in Python here which code could maybe be reused for this:
https://github.com/connectIOT/iottoolkit/

Hedda<EMAIL_ADDRESS>writes:

FYI, Pimporoni posted step-by-step instructions for how pne could
implement @hvanderlaan & @sandyjmacdonald Phyton code here:
https://learn.pimoroni.com/tutorial/sandyj/controlling-ikea-tradfri-lights-from-your-pi

Cool!  Is
this required for Trådfri (since commit dc91eb9 from Apr 12, you can
invoke coap-client without -u but I have no hardware to test against...)

Hi all,
I hope it is ok that I ask the following question here.
It is a little bit off topic since it has nothing to do with the ha-bridge but I think this is a great thread for questions regarding IKEA's Trådfri API.
I have made a small shell script but it does not work.
The echo output looks good and if I copy and paste the echo output manually in a terminal window it will work.
I suspect that the error is caused by some wrongly escaped quote that is not shown with echo but I am new to shell scripts so I need some advice.
Thanks in advance!
tradfri.sh
#!/bin/bash
set -x #echo on
exec coap-client -m put -u "\"Client_identity"\" -k "\"$1"\" -e ''\'{ "\"3311"\": [{ "\"5850"\": $2 }] }''\' "\"coaps://$3:5684/15001/$4"\"
exit

echo output & response from the coap-client after running the shell script:
$ sh tradfri.sh KEY 1 <IP_ADDRESS> 65537
+ exec coap-client -m put -u "Client_identity" -k "KEY" -e '{ "3311": [{ "5850": 1 }] }' "coaps://<IP_ADDRESS>:5684/15001/65537"
v:1 t:CON c:PUT i:dfb1 {} [ ]

@gobo-ws You have way too many quotes.
For example "\"Client_identity"\" translates to the string "Client_identity" being send, while is should just be Client_identity.

@r41d thanks!
Do not know why I made it harder than it was...
The following line works fine:
coap-client -m put -u "Client_identity" -k "$1" -e '{ "3311": [{ "5850": '$2' }] }' "coaps://$3:5684/15001/$4"

Hi, My colleagues and I have tried all of the above methods, involving several builds of libcoap with dtls, as well as using golang implementations of DTLS over COAP (https://github.com/zubairhamed/canopus) but have had no luck. Have IKEA changed something, we simply cannot get a key/certificate exchange to occur, as shown by the following wireshark dump,
coap-client -m put -u "Client_identity" -k "QkBSphFatfFs5VXq" -e '{ "3311": [{ "5850": 0 }] }' "coaps://<IP_ADDRESS>:5684/15001/65537"
v:1 t:CON c:PUT i:78d5 {} [ ]
^ No further output is produced
and

@csjames have you tried installing pytradfri by ggravlingen and test as a proof-of-concept?
https://github.com/ggravlingen/pytradfri
Follow the instructions in its README.md you install libcoap fork from home-assistant repo.

Ikea has already said they are adding support for the echo, Google home, and homekit in a month...

Homebridge

Ikea has already said they are adding support for the echo, Google home, and homekit in a month.
@Matt8119 that does not help it emulate Philips Hue API which is what ha-bridge does.

What ha-bridge does is to emulate Philips Hue API because it brings compatibility with many more home automations systems since most of them are already compatible with the Philips Hue API.
Samsung SmartThings => ha-bridge => Ikea Trådfri Gateway => Ikea Trådfri Lightbulbs/Lightpanels.
Vera => ha-bridge => Ikea Trådfri Gateway => Ikea Trådfri Lightbulbs/Lightpanels.
Harmony Hub => ha-bridge => Ikea Trådfri Gateway => Ikea Trådfri Lightbulbs/Lightpanels.
Once ha-bridge gets native support for Ikea Trådfri Gateway then you can use ha-bridge as bridge to other systems that does not yet have native support for Ikea Trådfri Gateway, such as Samsung SmartThings, Vera, Harmony Hub, and many more that already supports Ikea Trådfri Gateway.

 Turned out that the key printed in text was wrong, turned out that the QR code had an "o" instead of a "c", might have been scraped off but it was not easy to detect. Scan the QR code with a cell phone just to be sure.

Last night we reset our Tradfri hardware by holding the reset button down and then resetting all settings within the application.
This lead to the coap working fine, then, after about 10 minutes the base station proceeded to update and the coap interface became unresponse (5 second delays) for both the apps and coap-client. Following the update, performance returned to normal and everything is working smoothly.
Thanks for all your help! :)

Just added support for Moods in the MQTT bridge https://github.com/r41d/TRADFRI2MQTT/commit/eb90ef40f6a69b09ab05e080e6462d5435462686
Does anyone know if there is a DTLS-capable Python Library/binding yet?
All I've seen yet is this ugly calling of coap-client via subprocesses, this kinda keeps me from using anything else than Java atm :/

I need a libcoap implementation with DTLS support for iOS. It seems nothing is available... any idea ?

Anyone aware of a polling command to update the reachable property? I want to know if a bulb is reachable. The problem is that it seems like this property is not updated until something is commanded without success. If I disconnect a bulb, the property is not updated in the gateway (according to pytradfri), also not when I check the status from the app. But if I try to command something it is updated after some time. For the cheaper lamp with fixed colour temperature I can send a command to change colour temperature as a "NOP" command, reachable is then updated. Don't know what to send  to the other bulbs though, unless I save the commanded values and send them a again. Is there no "poll" command? Why doesn't the gateway poll the bulbs at a regular interval?

@hanpal Maybe you could try sending a command with invalid payload (out of range color XY values for example). In theory that would work the same like the color setting request on bulbs that don’t support it.

On 12 Jul 2017, at 10:27, hanpal<EMAIL_ADDRESS>wrote:
Anyone aware of a polling command to update the reachable property? I want to know if a bulb is reachable. The problem is that it seems like this property is not updated until something is commanded without success. If I disconnect a bulb, the property is not updated in the gateway (according to pytradfri), also not when I check the status from the app. But if I try to command something it is updated after some time. For the cheaper lamp with fixed colour temperature I can send a command to change colour temperature as a "NOP" command, reachable is then updated. Don't know what to send to the other bulbs though, unless I save the commanded values and send them a again. Is there no "poll" command? Why doesn't the gateway poll the bulbs at a regular interval?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Nice idea but didn't work. Invalid values stopped by gateway or pytradfri. Had no effect.

After reading this thread it seems like we now know which commands we need to send but there is still no ha-bridge support. Is this correct?
If so is anybody working on the tradfri integration with ha-bridge or ha-bridge to homebridge automation?
Last question. With ha-bridge is it possible to use the philips hue app and control your light from abroad?
Thanks!

I think this project needs more active developers other than bwssystems. One man can't do anything alone.
To your last Question regarding "control from abroad": No, because that would require to link ha-bridge with a hue account which is not possible. You can use the hue app with ha-bridge, but only locally.

To your last Question regarding "control from abroad": No, because that would require to link ha-bridge with a hue account which i believe is not possible. You can use the hue app with ha-bridge, but only locally.

To be fair, any person with enough knowhow to get ha-bridge running and working correctly is likely also able to configure a VPN server on their home router, in which case yes, yes you can control your lights remotely. (Though, I personally dont see the point in doing so, as we are all trying to build smart homes, the home should control the lights itself based on a variety of factors - having to remember to turn the lights on, off or dim to a specific setting while you're sitting on the playa with an umbrella drink isn't really a viable solution to this)

I've added support for changing colour temperature in whole spectrum between 2200K and 4000K: https://github.com/ggravlingen/pytradfri/pull/48/files
Next step: add it in home assistant.

@kloknibor Can't you just use the pytradfri solution?

Apparently E27 RGB bulbs are now available, today I bought one in an IKEA in Germany.
I didn't find anything online, so these gotta be released brand-new.
Here are some photos and screenshots: http://imgur.com/a/yC5VW
The Android App is able to display the colors that the bulb shows, but it's not able to set them. Setting RGB color only work ATM by repeatedly pressing the < and > buttons on the Tradfri remote.
Will try to do further investigation soon.

As soon as I get RGB bulb I'll be able to add support to pytradfri. @r41d , do you have IKEA product number (8 digits)? I'll check when they'll be available in Poland.

@r41d, what was the price of the RGB bulb compared to the 980lm white ones?

I've been having issues with my gateway and now cannot connect to it using coap-client. It was running firmware 1.1.0012 (i think) when i could connect but this lost connection to all the bulbs, which made it useless. I reset it and now it is running fireware version 1.0.008 according to the app. Bulbs are all connected however coap-client won't. Clicking check for updates says there are no updates. Any one had similar issues and know how to force it to update to 1.1.0015 from http://fw.ota.homesmart.ikea.net/feed/version_info.json

@matemaciek: on the back of the box it says 703.389.51
@emigrating: It cost 35€ for an RGB bulb with a remote (10€ more than a regular white spectrum bulb with remote) Sadly it wasn't possible to buy one without a remote... but I just had to get one ^^
@davetaz: also recently got problems with the coap-client connection to my gateway, damn, maybe they changed something.

@r41d I'm fairly confident my gateway has an issue at the moment as it won't upgrade. Will keep you posted.

If anyone wants to try most recent pytradfri with home-assistant, here it is https://github.com/home-assistant/home-assistant/compare/dev...matemaciek:dev

I made an attempt to document the Lwm2m+IKEA JSON format that TRÅDFRI communicates with. It is derived from these sources:

http://sprunge.us/CCQF
https://github.com/ggravlingen/pytradfri/blob/master/pytradfri/const.py
https://github.com/hardillb/TRADFRI2MQTT/blob/master/src/main/java/uk/me/hardill/TRADFRI2MQTT/TradfriConstants.java

And by querying my own Trådfri system with my own python script.
The format of the JSON structure below is my own invention so hopefully you can make sense of it...
  "15001": {
    "name": "DEVICES",
    "description": "Device ids for lights and remotes.",
    "example": [
      65536,
      65537,
      65538
    ],
    "15001/reset": {

    },
    "15001/{DeviceId}": [
      {
        "9019": {
          "name": "REACHABILITY_STATE",
          "example": 1
        },
        "3": {
          "name": "DEVICE_INFO",
          "example": {
            "1": "TRADFRI bulb E27 CWS opal 600lm",
            "0": "IKEA of Sweden",
            "3": "1.3.002",
            "2": "",
            "6": 1
          }
        },
        "5750": {
          "name": "APPLICATION_TYPE",
          "example": 2
        },
        "9003": {
          "name": "INSTANCE_ID",
          "example": 65538
        },
        "9002": {
          "name": "CREATED_AT",
          "example":<PHONE_NUMBER>
        },
        "9001": {
          "name": "NAME",
          "example": "Badass bulb"
        },
        "3311": {
          "name": "LIGHT_CONTROL",
          "example": [
            {
              "5708": 0,
              "5709": {
                "name": "LIGHT_COLOR_X",
                "example": 30140
              },
              "5710": {
                "name": "LIGHT_COLOR_Y",
                "example": 26909
              },
              "5706": {
                "name": "LIGHT_COLOR",
                "example": "f1e0b5"
              },
              "5707": 0,
              "5711": 0,
              "5712": {
                "name": "TRANSITION_TIME",
                "description": "Specified in seconds/10.",
                "example": 100
              },
              "5850": {
                "name": "ON/OFF",
                "example": 1
              },
              "5851": {
                "name": "LIGHT_DIMMER",
                "example": 160
              },
              "9003": {
                "name": "INSTANCE_ID",
                "example": 0
              }
            }
          ]
        },
        "9054": {
          "name": "OTA_UPDATE_STATE",
          "example": 0
        },
        "9020": {
          "name": "LAST_SEEN",
          "example":<PHONE_NUMBER>
        }
      },
      {
        "9019": {
          "name": "REACHABILITY_STATE",
          "example": 1
        },
        "3": {
          "name": "DEVICE_INFO",
          "example": {
            "1": "TRADFRI remote control",
            "0": "IKEA of Sweden",
            "3": "1.2.214",
            "2": "",
            "6": 3,
            "9": 87
          }
        },
        "5750": {
          "name": "APPLICATION_TYPE",
          "example": 0
        },
        "9003": {
          "name": "INSTANCE_ID",
          "example": 65536
        },
        "9002": {
          "name": "CREATED_AT",
          "example":<PHONE_NUMBER>
        },
        "9001": {
          "name": "NAME",
          "example": "TRADFRI remote control"
        },
        "9054": {
          "name": "OTA_UPDATE_STATE",
          "example": 0
        },
        "9020": {
          "name": "LAST_SEEN",
          "example":<PHONE_NUMBER>
        },
        "15009": {
          "name": "SWITCH",
          "description": "No references to this anywhere",
          "example": [{
            "9003": {
              "name": "INSTANCE_ID",
              "example": 0
            }
          }]
        }
      }
    ]
  },
  "15004": {
    "name": "GROUPS",
    "description": "Groups of devices.",
    "example": [
      154580
    ],
    "15004/add": {

    },
    "15004/remove": {

    },
    "15004/{GroupId}": {
      "5850": {
        "name": "ON/OFF",
        "example": 1
      },
      "9002": {
        "name": "CREATED_AT",
        "example":<PHONE_NUMBER>
      },
      "9001": {
        "name": "NAME",
        "example": "My phat group name"
      },
      "9003": {
        "name": "INSTANCE_ID",
        "example": 154580
      },
      "9039": {
        "name": "SCENE_ID",
        "example": 205512
      },
      "9018": {
        "name": "HS_ACCESSORY_LINK",
        "example": {
          "15002": {
            "name": "HS_LINK",
            "example": {
              "9003": {
                "name": "INSTANCE_ID",
                "description": "DeviceIds reached by: 15001/DeviceId",
                "example": [
                  65536,
                  65537,
                  65538
                ]
              }
            }
          }
        }
      },
      "5851": {
        "name": "LIGHT_DIMMER",
        "example": 0
      }
    }
  },
  "15005": {
    "name": "SCENES_TOP",
    "description": "Some kind of top scene structure",
    "example": [
      154580
    ],
    "15005/{SceneTopId}": {
      "name": "SCENES",
      "description": "List of all scenes",
      "example": [
        217404,
        228154,
        203200,
        205512
      ],
      "15005/{SceneTopId}/{SceneId}": {
        "9068": {
          "name": "IKEA_MOODS",
          "example": 1
        },
        "15013": {
          "name": "LIGHT_SETTING",
          "description": "Interesting that two LIGHT_CONTROL's are specified here.",
          "example": [
            {
              "5708": 0,
              "5709": {
                "name": "LIGHT_COLOR_X",
                "example": 30140
              },
              "5710": {
                "name": "LIGHT_COLOR_Y",
                "example": 26909
              },
              "5706": {
                "name": "LIGHT_COLOR",
                "example": "f1e0b5"
              },
              "5707": 0,
              "5711": 0,
              "5850": {
                "name": "ON/OFF",
                "example": 1
              },
              "5851": {
                "name": "LIGHT_DIMMER",
                "example": 203
              },
              "9003": {
                "name": "INSTANCE_ID",
                "example": 65537
              }
            },
            {
              "5708": 0,
              "5709": {
                "name": "LIGHT_COLOR_X",
                "example": 30140
              },
              "5710": {
                "name": "LIGHT_COLOR_Y",
                "example": 26909
              },
              "5706": {
                "name": "LIGHT_COLOR",
                "example": "f1e0b5"
              },
              "5707": 0,
              "5711": 0,
              "5850": {
                "name": "ON/OFF",
                "example": 1
              },
              "5851": {
                "name": "LIGHT_DIMMER",
                "example": 203
              },
              "9003": {
                "name": "INSTANCE_ID",
                "example": 65538
              }
            }
          ]
        },
        "9003": {
          "name": "INSTANCE_ID",
          "example": 217404
        },
        "9002": {
          "name": "CREATED_AT",
          "example":<PHONE_NUMBER>
        },
        "9001": {
          "name": "NAME",
          "example": "EVERYDAY"
        },
        "9057": {
          "name": "SCENE_INDEX",
          "example": 0
        }
      }
    }
  },
  "15006": {
    "name": "",
    "example": [{
      "9014": {
        "name": "NOTIFICATION_STATE",
        "example": 0
      },
      "9015": {
        "name": "NOTIFICATION_EVENT",
        "example": 5001
      },
      "9002": {
        "name": "CREATED_AT",
        "example":<PHONE_NUMBER>
      }
    }]
  },
  "15010": {
    "name": "SMART_TASKS",
    "example": []
  },
  "15011": {
    "name": "GATEWAY",
    "description": "Operations on the gateway",
    "15011/15012": {
      "name": "GATEWAY_DETAILS"
    },
    "15011/9034": {
      "name": "UPDATE_FIRMWARE"
    },
    "15011/9030": {
      "name": "REBOOT"
    },
    "15011/9031": {
      "name": "RESET"
    },
    "15011/9063": {
      "name": "AUTH_PATH"
    },
    "15011/9033": {
      "name": "SESSION_ID"
    }
  },
  "status": {
    "name": "",
    "example": {}
  }
}

@r41d managed to get my gateway working. It would seem that the gateway might hold two firmware images and each time you reset the gateway it boots either the factory default or the latest one it has. So I had my gateway running 1.1.0015 but it was corrupt, so I reset and the gateway booted 1.0.008 and it remembered the devices. Although the gateway was running 1.0.0008 the updates screen said the gateway was up to date. This is because it was, I just wasn't running that image. I then reset it again and the gateway booted a working 1.1.0015 and forgot all my devices. After re-adding them everything is now fine and homebridge and coap is all happy.

Seemingly, IKEA released the first update to the Tradfri Android App.
One can now choose some colors when tapping on a bulb and then on the three dots: https://imgur.com/a/0A7iq
These are many more than those colors which one can cycle through by using the physical remote

I'm gonna get my RGB bulb finally today, I know that they're in my IKEA (-:

Someone know how to change color for an entire group ? I tried to change the mood but I didn't want to change my dimmer level.

@Hydro8 I'm also interested in that

Hello guys, just wanna post my experience with development version of libcoap and some issues I have with it.
If you git clone libcoap and then checkout devel branch, you get v4.2.0alpha which is way forward of the latest stable in master (4.1.2) with a bunch of bug fixes etc.
Since I'm developing my own Tradfri coap-mqtt bridge in python2, I use coap-client extensively (no support for python yet and py3 version is incomplete) in polling mode, and I've noticed that it has more extensive output on stdout than before.
One line more, to be precise, where it writes additional info about data type it handles, so take care of that in your parsing programs.
But, one more thing that still fails, like in old version, is subscription, or observe.
It appears that you get Observe:0 option when using -s switch to coap-client which should emulate the usual mqtt subscribe method. Now, in coap-client it works for some time, but after some undetermined time it breaks and the client unsubscribes itself - or the server (Tradfri hub, in my case) kicks it out.
Playing with -s timeout doesn't really change anything, and I don't even understand what is exactly that timeout for.
So, my question - did somebody manage to effectively subscribe to an endpoint (topic) on coap server without this disconnecting behaviour?
If the problem is in coap-client, does anybody have and other cli coap client that actually works with tradfri?
I can't compile LibNyoci on multiple platforms, and FreeCOAP has no usable client and no DTLS, so...bummer.

@UserXYZ I believe it's an issue with libcoap, as we don't experience the issue with aiocoap over at ggravlingen/pytradfri

Well, I guess then that the developer of libcoap made a bug that needs to be fixed, or he chose such behavior of coap-client deliberately, since that is supposed to be a test tool...
What surprises me is that no other contributor decided to fix it, or at least make subscribe parameter with value of zero work as normal Observe:0 option, which would actually be a regular no-timeout subscription.
Since I can't use Python 3 (still being under much of development etc), aiocoap is not a solution atm...
So, anybody knows of another software that can be used instead of libcoap? Should be written in C/C++ (simplest to build), have dtls using tinydtls (gnutls is pain to include and I'm not sure if it works with keys besides certs) and pretty much everything else that libcoap has...?
Like a better version of libcoap...

@UserXYZ You are always welcome to open issues on the libcoap bugtracker or discuss these on the libcoap mailing list. It is difficult to fix your issues if nobody knows about them.
You can adapt the amount of output produced by coap-client on stdout with the option '-v'.
Regarding the timeout: '-B' controls the time until coap-client terminates. By default, this is 90 seconds as documented in the manual page. The option -s controls the timeout until coap-client cancels the established observe relationship.
Regarding your question about alternatives to libcoap: There is Californium which has DTLS support, and AFAIK, there is a version of node-coap with DTLS support.

I came a cross a similar problem : Getting something not supported by ha-bridge working with ha-bridge. (Denon Amplifier, Samsung TV, 433Mhz based HA kit and more.
Rather than delving into ha-bridge's code, I ended up writing my own php broker and ha-bridge just flings the http command at it and the broker sorts out the rest.
Alexa / Google => ha-bridge => My Broker => Insert Weird Gateway => Weird Kit here.
It appears from the above a similar approach is being attempted by some of you. The work carried out by Adreas Spiess may help : https://www.youtube.com/watch?v=wS8Vj0ba4ic.

@davetaz I have exactly the same problem, my gateway is stuck on 1.0.0008 and says there are no updates. I tried reseting with the hole button but it stays in the same version. How did you make it to boot to the other image?

Here are some of the things I did.

Keep trying the reset
Each time you do, also reinstall the app on your phone
Take the gateway back to IKEA (which is what I did in the end)

@lwis can you tell me how do you use aiocoap with ggravlingen/pytradfri code? In asynchronous mode, I suppose, because in synchronous mode it is just calling coap-client in the background (saw that in my process list) which kinda defeats the purpose of aiocoap...

@UserXYZ guts are all here buddy.

@lwis Well, that didn't quite answer my question, but based on your link, I'd say async mode...
So, no problems with observation of resource, you can track it's state for as long as you like?
No need to poll, or at least disconnect and reconnect at some time to give the hub a chance to clean connections or something?

@UserXYZ Oh, you mean observation? Yes, coap-client has issues with observation, but aiocoap does not. You can observe until the connection is stopped at either end.

@lwis Sweet, that's what I need...now just to figure out aiocoap's API to reimplement my code or switch totally to pytradfri as a base...oh, and learn Python3 alongside Python2...such fun...

@UserXYZ obviously I'd recommend using pytradfri 😉

@lwis Weeell, that would be easier way to do it, for sure, but again it's also adding another layer.
my code->pytradfri->aiocoap and I could do my code->aiocoap but with more effort on my part...
Meh, I'll try with pytradfri first and then see if I get the hang of aiocoap's API, make another branch/version then.

I found my GW, but it is not open on port 5684
The gateway is running version 1.2.42, claiming to be hue-compatible.
I'm currently scanning all ports; none seem to be open.

With 1.2.42 IKEA changed the authentication. See the following link: https://community.openhab.org/t/ikea-tradfri-gateway/26135/148?u=kai

Where do you find the coap-client?
I got the one from libcoap-1-0-bin in debian.
It does not have the -u option.
`
root@mini:# coap-client -m post -u "Client_identity" -k "qwertyuiopå" -e '{"9090":"nodered"}' \
"coaps://<IP_ADDRESS>:5684/15011/9063"

coap-client: invalid option -- 'u'
coap-client v4.1.2 -- a small CoAP implementation
   (c) 2010-2015 Olaf Bergmann<EMAIL_ADDRESS>
usage: coap-client [-A type...] [-t type] [-b [num,]size] [-B seconds] [-e text]
            [-m method] [-N] [-o file] [-P addr[:port]] [-p port]
            [-s duration] [-O num,text] [-T string] [-v num] [-a addr] [-U] URI

`
I'm trying to access the hub from node-red, and it will not authenticate.
node-red-contrib-tradfri 1.1.0

Registration did not work. Verify your settings and possibly try another identity. Note that an identity can only be used once and the response preshared key must be stored for usage with the identity

For testing of the gateway purpose i'm using the californium.tools (https://github.com/eclipse/californium.tools) and in my program i use the coap client implementation  of the californium library.

Hi, I've been following this (nice) discussion from the very beginning. The thing is that I would be interested in running some (source) code directly (by passing the gateway) on the bulbs. Do you think it would be possible? (a piece of software running continuously in the bulb... if the temperature of the bulb falls down, then change the color to blue)

Hello! Tons of useful info and links in the thread, thanks! I have been playing with the new Trådfri Color trying to figure out why it would not show the whole spectrum and found the answer by cracking one open. It does not have green LEDs. I have put some code and info here: https://github.com/ffleurey/ThingML-Tradfri so that you do not have to vut your own bulb :-) Cheers!

@ffleurey Really nice library. Great that you have put the effort in to make the UI, really fun and works great.

Very informative thread.  I'm trying to generalize some of the findings above into a handy CoAP Shell (https://github.com/tzolov/coap-shell) with some initial Ikea gateway support as well.
https://www.youtube.com/watch?v=zhEGFfCJwTg&feature=youtu.be

With the implementation of OpenHAB, Which has Tradfri connectivity, I will suggest that is the path to implement this.
See https://www.openhab.org

New features of GitHub are really cool. It's more convenient to use now than before. You can make cool custom boxes website in no time.

^ Please report @SamSpringPackaging for spam (go to the profile and click 'Block or report user')

@vandenberghev Thanks! He is reported and banned from BWS Systems


feat(github_enterprise): Allow use of GitHub Enterprise

Proposed update for #16

I like this, but I think we can get a default from the environment in the form of GITHUB_SERVER_URL. I'll see about adding that.

Closed in favor of https://github.com/andrewthetechie/gha-repo-manager/pull/23


System clipboard

fix https://github.com/excalidraw/excalidraw/issues/859
Quickly hacked together, not sure if correct. Play with it and report, esp on browsers that don't support async clipboard API (firefox, safari?), and copy/paste between tabs/browsers, using various APIs (keyboard cmd-c/v, contextmenu...).
I'll tighten up the code and write tests later.

I could paste from all browsers to Chrome, Safari, but no Firefox.. could copy stuff from Firefox to other browsers though! Cool feature.

I could paste from all browsers to Chrome, Safari, but no Firefox.. could copy stuff from Firefox to other browsers though! Cool feature.

Cmd-c from chrome and then Cmd-v to Firefox doesn't work? AFAIK the only thing that shouldn't work is Cmd-c from other browser and contextmenu-paste to Firefox.

I didn't use the context menu.. just cmd+C/cmd+v.. out of the office now, so can't do more tests.

Also note that I changed the schema of the object, so if you copied from a production and pasted into preview (or vice versa), then it wouldn't work.

It's a bit buggy when using keyboard shortcuts for copy/pasting between tabs/browsers.. but works just fine when using the CM

It's a bit buggy when using keyboard shortcuts for copy/pasting between tabs/browsers

Any specifics with steps would be appreciated!

@lipis @Fausto95 gentle reminder, if you have reproduction steps on any bugs you have encountered with the copy/paste, please share

@dwelle on Safari, after drawing some elements, when you try copy with keyboard shortcuts then try to paste it in another tab or even in Chrome, it doesn't work. But if you do it with CM, it works just fine.
After copying/pasting with CM the first time, doing it using KS it works.

It seems that it works from all browsers that I tested (Chrome, Safari, Firefox)
I was using the shortcuts.

After copying/pasting with CM the first time, doing it using KS it works.

Oh, ok.. that sounds like the permission thing again. I'll try to investigate. Thanks.

@Fausto95 I wasn't able to reproduce it across chrome/firefox tabs/browsers. The closest I could go is try to invoke the clipboard permission prompt (via contextmenu pasting) enough times without accepting that the browser automatically blocked further attempts. At that point I lost the ability to paste via contextmenu (until I manually reenabled the permission).
But what you're seeing I couldn't reproduce at all because keyboard shortcut pasting is preferentially handled via native event.clipboardData.getData() API which has no permissions attached. Not sure what else I can do.
As for single-tab copy/pasting, the above shouldn't happen at all (but it seems even you didn't encounter that) because the last fallback is in-memory clipboard.
If the issues happen only cross-tab/cross-browser, I think it's safe to go forward with this.

I think we can merge and improve later!

I'll review and clean up the code, and potentially write tests. Will merge later today.

Let's merge this with current state.. and if users start to complain, will fix it


Recorded Future for Splunk SOAR: Bug-fix - Release of app version 4.3.2

Please ensure your pull request (PR) adheres to the following guidelines:

Please refer to our contributing documentation for any questions on submitting a pull request, link: Contribution Guide

Pull Request Checklist
Please check if your PR fulfills the following requirements:

[x] Testing of all the changes has been performed (for bug fixes / features)
[x] The manual_readme_content.md has been reviewed and added / updated if needed (for bug fixes / features)
[x] Use the following format for the PR description: <App Name>: <PR Type> - <PR Description>
[x] Provide release notes as part of the PR submission which describe high level points about the changes for the upcoming GA release.
[x] Verify all checks are passing.
[x] Do NOT use the next branch of the forked repo. Create separate feature branch for raising the PR.
[x] Do NOT submit updates to dependencies unless it fixes an issue.

Pull Request Type
Please check the type of change your PR introduces:

[ ] New App
[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no api changes)
[x] Documentation
[x] Other (please describe): Copyright notice date update

Security Considerations (REQUIRED)

[x] If you are exposing any endpoints using a REST handler,
please document them in the manual_readme_content.md.
[ ] If this is a new connector or you are adding new actions

[ ] Please document in the manual_readme_content.md all methods (eg, OAuth) used to authenticate
with the service that the connector is integrating with.
[ ] If any actions are unable to run on SOAR Cloud, please document this in the manual_readme_content.md.

[ ] Are you introducing any new cryptography modules? If yes, please elaborate their purpose:
[x] Are you are accessing the file system? If yes, please verify that you are only accessing paths returned through
the Vault API.
[x] Are you are marking code to be ignored by Semgrep with nosemgrep?
If yes, please provide justification in an additional comment next to the ignored code.
API key fingerprint is logged for debugging purposes.

Release Notes (REQUIRED)

Provide release notes as part of the PR submission which describe high level points about the changes for the upcoming GA release.

What is the current behavior? (OPTIONAL)

Describe the current behavior that you are modifying.

What is the new behavior? (OPTIONAL)

Describe the behavior or changes that are being added by this PR.

Improved visibility of support documents
Renaming of app headers
Fixing issues with hardcoded path for Cloud
Improved format for Intelligence Command Widgets
Added status config options for fetching standard and playbook alerts

Other information (OPTIONAL)

Any other information that is important to this PR such as screenshots of how the component looks before and after the change.

Pay close attention to (OPTIONAL)

Any specific code change or test case points which must be addressed/reviewed at the time of GA release.

Screenshots (if relevant)

Thanks for contributing!

Hi @dhwanis-crest
Looking to get this merged, but I'm a bit confused about the AWS CodeBuild us-west-2 (prodsec-scans) action that appears to be failing. Clicking details redirect me to an AWS login page.

@nestoor22 suggested that I reach out to you.
Do you have any more details as to why this is failing?
Thanks!

The prodsec fail is caused by requests package, please remove it from requirements, as it is supplied by SOAR platform @JQvickWester

@bb-splunk Thank you for you assistance, that cleared it up - have a great day!
@nestoor22 looks like this is ready to merge, if you'd be so kind!


Update view-analytics.md

Can the article include the note that this isn't available to users in the DoD tenant? I opened a case with Microsoft back in October and they mentioned that it wasn't available to users in the DoD and would like to have this article updated to share with users.

Microsoft Case # for reference:<PHONE_NUMBER>009546


mimxrt1050_evk: Support MCU Boot using Hyperflash

Describe the bug
Build samples/subsys/usb/dfu/ for mimxrt1050_evk fails
To Reproduce
Steps to reproduce the behavior:
1.west build -b mimxrt1050_evk zephyr/samples/subsys/usb/dfu/ -d build-dfu
Expected behavior
The build will fail when building the flash_img object.
Logs and console output
In file included from /home/user/zephyrproject/zephyr/include/toolchain.h:50,
                 from /home/user/zephyrproject/zephyr/lib/libc/minimal/include/stdio.h:12,
                 from /home/user/zephyrproject/zephyr/subsys/dfu/img_util/flash_img.c:10:
/home/user/zephyrproject/build-dfu/zephyr/include/generated/devicetree_unfixed.h:41208:50: error: 'DT_N_S_soc_S_spi_402a8000_S_s26ks512s_0_P_write_block_size' undeclared here (not in a function); did you mean 'DT_N_S_soc_S_spi_402a8000_S_s26ks512s_0_P_cs_hold_time'?
41208 | #define DT_CHOSEN_zephyr_flash                   DT_N_S_soc_S_spi_402a8000_S_s26ks512s_0
      |                                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/user/zephyrproject/zephyr/include/devicetree.h:562:32: note: in expansion of macro 'DT_CAT'
  562 | #define DT_PROP(node_id, prop) DT_CAT(node_id, _P_##prop)
      |                                ^~~~~~
/home/user/zephyrproject/zephyr/subsys/dfu/img_util/flash_img.c:33:2: note: in expansion of macro 'DT_PROP'
   33 |  DT_PROP(DT_CHOSEN(zephyr_flash), write_block_size)
      |  ^~~~~~~
/home/user/zephyrproject/zephyr/include/devicetree.h:2991:24: note: in expansion of macro 'DT_CHOSEN_zephyr_flash'
 2991 | #define DT_CAT(a1, a2) a1 ## a2
      |                        ^~
/home/user/zephyrproject/zephyr/include/devicetree.h:1853:25: note: in expansion of macro 'DT_CAT'
 1853 | #define DT_CHOSEN(prop) DT_CAT(DT_CHOSEN_, prop)
      |                         ^~~~~~
/home/user/zephyrproject/zephyr/subsys/dfu/img_util/flash_img.c:33:10: note: in expansion of macro 'DT_CHOSEN'
   33 |  DT_PROP(DT_CHOSEN(zephyr_flash), write_block_size)
      |          ^~~~~~~~~
/home/user/zephyrproject/zephyr/subsys/dfu/img_util/flash_img.c:35:43: note: in expansion of macro 'FLASH_WRITE_BLOCK_SIZE'
   35 | BUILD_ASSERT((CONFIG_IMG_BLOCK_BUF_SIZE % FLASH_WRITE_BLOCK_SIZE == 0),
      |                                           ^~~~~~~~~~~~~~~~~~~~~~
/home/user/zephyrproject/zephyr/subsys/dfu/img_util/flash_img.c:35:14: error: expression in static assertion is not an integer
   35 | BUILD_ASSERT((CONFIG_IMG_BLOCK_BUF_SIZE % FLASH_WRITE_BLOCK_SIZE == 0),
      |              ^
[65/192] Building C object zephyr/CMakeFiles/zephyr.dir/subsys/usb/device/usb_device.c.obj
ninja: build stopped: subcommand failed.
FATAL ERROR: command exited with status 1: /usr/bin/cmake --build /home/user/zephyrproject/build-dfu

If applicable, add console logs or other types of debug information
e.g Wireshark capture or Logic analyzer capture (upload in zip archive).
copy-and-paste text and put a code fence (```) before and after, to help
explain the issue. (if unable to obtain text log, add a screenshot)
Environment (please complete the following information):

OS: Ubuntu 20.04
Toolchain zephyr-sdk-0.14.0
Commit e97a2ee6d0d64a3c1711942db854e6ccf78ea8d6

Hi @kunoh, MCUBoot (and therefore USB DFU mode) is not currently supported on the RT1050 EVK using the hyperflash, because MCUBoot does not support flash devices with a minimum write size of 16 bytes, only those with a minimum write alignment of 8 or smaller. If you use the QSPI flash (see AN1208 for for information), you should be able to use MCUBoot. I'm going to go ahead and mark this as an enhancement (and update the title), please let me know if the new title does not adequately reflect the issue.

Hi @danieldegrasse, I changed my board to use QSPI and it works with MCUBoot, thank you - QSPI is also an excluded platform for the DFU example. Do you know why and what's needed in order to get this to work? (Sorry, maybe the question belongs to another issue?)

I tried this sample on a few other evaluation kits, and the issue seems to be that the FlexSPI flash is too large to erase in one step (doing so results in USB communication timing out). Enabling CONFIG_IMG_ERASE_PROGRESSIVELY fixed the issue for me. @kunoh, can you try #45358 on the RT1050 modified to use QSPI flash, and see if device firmware upgrade works for you?

Thank you @danieldegrasse, it works when CONFIG_IMG_ERASE_PROGRESSIVELY is enabled!


Stopped Working?

Extension Version: 1.4.0 (May 21, 2019)
Browser: Firefox 67.0.4 and Chrome 75.0.3770.100
Operating System: Windows 10 (Version 1903)
URL of the problem page: https://medium.com/flutter-community/a-deep-dive-into-stack-in-flutter-3264619b3a77

Error on the devtools (not sure if it coming from this extension)

Hi @apiep!
Hmm, no it's not those errors causing the problem. It seems that in Firefox, the <head> doesn't include the attributes that the addon is targeting:
<head prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb# medium-com: http://ogp.me/ns/fb/medium-com#">

What's really weird is if I view-source on that page in Firefox vs Chrome, the HTML is completely different.

Ugh, I doubt I'll make this work... the content is "dynamically" loaded in Firefox; which I can get around; but then I realized that all the class names are useless:

Feedback I just sent to Medium:

Hi, I'm the author of the Darker-Medium browser (Chrome, Firefox & Opera) extension
(https://github.com/Mottie/Darker-Medium). Right now if I go to this page on my desktop
computer: https://medium.com/flutter-community/a-deep-dive-into-stack-in-flutter-3264619b3a77 -
the source HTML in Firefox is completely different than in Chrome (I didn't check any other
browsers). Firefox is being made to dynamically load all its content (why on a desktop?) and
thus it ends up with class names like "jp", "ec" & "kc". These classes also appear on all @
user profile pages. Please please please use some reasonable css class names on these
pages. Or, provide a dark theme that we can switch to when desired. Oh, and please tell
the designers/developers to not use !important flags in their css everywhere. It makes it
very difficult to create/update/apply style overrides to Medium story pages.

Response from Medium:

Hi,
We are migrating our pages to a new stack, and this could be a result of that. (and the discrepancies you see could be a result of viewing the pages in different roll-out cohorts).
I will raise this issue to the team to investigate. Please bear with us.

It seems, they decide to use "modern frontend". That is why it's css selector are minified and I see a giant minified object when I try to inspect it source too.
And I am not sure what we can do about it except manually choose the correct element to style.


access /metrics says 404 not found when config it by adding "prometheus:          enabled: true          path: "/metrics""

Closing as working as expected in v3


Move useStatefulResource into the library itself

Is your feature request related to a problem? Please describe.
The useStatefulResource hook is not that big, can probably be tree-shaken, and feels like it belongs in rest-hooks itself. The reason I'm asking this is that the types seem to break periodically, and importing my own implementation feels kind of weird.
(Don't worry: I'm actually using this for stuff that doesn't make too much sense with suspense, like showing loading states in tables)
Describe the solution you'd like
Make useStatefulResource an export from 'rest-hooks'

What if this was released as another package like @rest-hooks/legacy

Would be happy with that as well

https://resthooks.io/docs/guides/no-suspense has been updated to refer to the new package @rest-hooks/legacy


potential renaming of preset "Recently demolished building"

Potential follow up of #7098
Preset of tag: demolished:building=*
Proposal:
Renaming of preset "Recently demolished building." to "Outline of recently demolished building." or something similar, e.g. more meaningful adjusted to native English speaker.
btw.: Are there also presets available for:?

removed:building=* - https://taginfo.openstreetmap.org/keys/removed%3Abuilding ~ 8k db-units
razed:building=* - https://taginfo.openstreetmap.org/keys/razed%3Abuilding ~ 3k db-units

Are there also presets available for:?

Nope, since the distinction between different types of former buildings isn't really significant in OSM. The point of the demolished:building preset is to prevent mappers from adding former buildings that are still visible in imagery. It would be nice if these were all tagged the same way.

It's true - the distiction is hard, it seems more or less these are the same kind of objects - but it also seems the community didn't clearly deside (there is no approved proposal so far i see) which variant is to use. So therefore i would like to have available at least for the top two (demolished:building=* & removed:building=*) the same hidden preset. (http://taghistory.raifer.tech/#/removed:building/&/demolished:building/&***/razed:building/)

So therefore i would like to have available at least for the top two (demolished:building=* & removed:building=*) the same hidden preset.

iD can't have one preset that matches multiple different tags. There's also no precedent for having multiple presets for the same thing. If the OSM community settles on one tag, then iD can support that and deprecate the others, but until then we're just going with the most-used pattern.

Oh sorry, of course, it was a misunderstanding from my side. Was maybe triggered because i see a maybe minor difference between demolished: (i thought it might be destroyed and maybe some rests of the destroyed parts are still there) and removed: (i thought all parts are completely removed). But you are right, about this should be talked first by a wider community.


Embeddings from the trained model don't work.

Hey, I loaded your pretrained model for CIFAR-10 using the code given in the readme, and then simply tried performing inference on my own with the model. For some reason, I'm getting very poor accuracy metrics (11%) on the CIFAR 10 test set.
from torchvision.datasets import CIFAR10

dataset_dir = "/content/"
mean = [0.4914, 0.4822, 0.4465]
std = [0.2023, 0.1994, 0.2010]
transform_val = transforms.Compose([transforms.ToTensor(), transforms.Normalize(mean, std)])
dataset = CIFAR10(root= dataset_dir, train=False, transform=transform_val,  download = True)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=128, num_workers=4, pin_memory=True)

Then I perform simple inference with y=model(x) format to get the embedding, and I argmax the 10-dim values I get to obtain my predicted class values.
Also when I normalise, most of the values just turn out to be class 1 after I argmax. Very weird.
Am I doing something wrong here? I get 11% accuracy doing this.

Have you put your model in eval mode? Please try again with the new version.

Have you put your model in eval mode? Please try again with the new version.


Do a release because git submodules are weird.

Closed by 8c9a2605
Thanks for the feedback, @younata.
Would you be interested in using this with Alcatraz? I've been thinking about investigating what would be required to get it installed through that plugin manager...