error in operatingData / map request [504 / 500/ 429]

Version: 0.9.12
Indego: M700+ (2020)
Firmware: 27176

500 and 504 are Error Codes where the service did not respond correctly. This could have different reasons. E.g. connection problems, server errors etc.
The Adapter should check periodicaly and try to reconnect.
Did id reconnect by it self after some time?
There is not much i can do about this, exect .. not show this message :)

Ok then it is a problem within the Bosch API / Server - their servers are having problems regularily at the moment ;)
Maybe you could edit the log message to something like: "Bosch server responded with error code 500" then it would be clear.
Other than that your adapter works like a charm! No problems with new firmware. Also tried ioBroker Admin 5.1.21 - works perfectly!

Thanks! Yes. I will make some moddification to the error logging.
I think it should try to recover by it self .. if that does not work .. only then the a error message should be shown.
Did it recover in your case after some time?
Did you see alot of this errors?

Didn't recover yet. But status codes and messages are still getting updated.

OK interessting! Thanks! How long did the errors show in total?

Several hours in total until now.
Just did a reboot of the machine running ioBroker. It seems to have fixed the errors:

In comparison to my logs above with the errors, there was no "bot stopped" immediately after "connect ok"

Uh. Ok. That s strange. I investigated further and made some changes. The Adapter should force a reconnect ( refresh auth etc ) if something like this happens. I pushed a new version with this changes.
Did this situation happen multiple times, or just this once? Would be nice if you could test with this version.

This message was shown when I logged into ioBroker just now (0.9.12).
Installing the new version now and testing. Thanks Markus!

Version 0.9.12

Installed Version 0.9.13. No errors so far. I'll keep reporting.

Version 0.9.13
Seems like the Bosch servers are heaving trouble...

Hmm .. very strange. Never got that during my extensive testing. 429 doesn t look to good.
What kind of connection do you have?
Could you try to test if there are problems with your internet connection
Open a commad prompt " click on windows logo and type "cmd" and start it.
There type:
ping google.de /t
This will ping the google website every seconde.
You can let this run for quite some time. And if you encounter more errors ( exept the 429 ) press "ctrl + c" or "strg + c"
After that you will see how many requests where dropped.

It will look like this.
How are you using the adapter? Do you open your app on your mobile often?
Does this run on multiple machines?

Yes, I'm opening the app very often recently.
Your Adapter runs on one machine only.
Did the ping test on that machine:

No packets lost, and no errors in ioBroker logs since the manual instance reload.
I recently installed Admin 5.1.21 Beta for ioBroker and did no other major changes.
It's the first time I encountered that problem.
I assume I produced too many requests to Bosch servers and maybe got some sort of softban perhaps.
I'll try to provoke it again and going to see what the ping test does while getting errors.

Yes, I'm opening the app very often recently.
I'm doing some testing and optimizations to the mower itself, so I'm constantly checking its position etc.
Therefore, in theory, I'm probably producing "unnatural" amounts of requests.

I don't know whats going on. No packet loss during that timeframe.
The adapter ran fine today until I started the mobile app, while being on the same ip as the machine running ioBroker.

I reinstalled 0.9.12.
While there are still some error 500 left, I don't get any 429 or force close of the adapter by iobroker.
I assume the immediate reconnects of 0.9.13 are too much load on the Bosch servers leading to error 429.

OK i made some changes and uploaded 0.9.14.
The reconnect will be handled more like in the version 0.9.12.
Still strange that you see this errors. I don t get any. Even if i use the mobile app.
Are you using android or ios?

Still strange that you see this errors. I don t get any. Even if i use the mobile app.
Are you using android or ios?

It's very strange..
Hasn't been a Problem before installing Admin 5, JS Controller 3.3 and Indegos new firmware 27176.
Obviously one of those three must have broken my personal setup.
Or my account at Bosch is flagged for some reason. Maybe I should contact them for clearance.
I use the Android app. Even with the App is sometimes get a blank map instead of my garden app, wich would add to the assumption that it's a server error on Boschs side.
I'm hoping this is not a waste of your time, but maybe that's still interresting for the error handling of the adapter?
With 0.9.14 I'm getting the same iobroker error as with .13:

The logs leading to that forced termination by iobroker:

I'm reading it that way:
Adapter gets 504, tries to reconnect immediately, followed by a 429 "softban" because of the many fast connection attempts.

Hmm .. Just to be sure.
When i update the Adapter in my system i:

stop the adapter
delete the adapter
go to the adapter List, and delete the adapter shown there
In Adapter List  click on gitub icon. Go to "Custom" ( Benutzerdefiniert ) and paste in the Adapter URL "https://github.com/a1development/ioBroker.boschindego" and click on install
Install the Adapter on the machine and enter credentials.
Start adapter.

I just tested this adapter on Admin5 with JS Control 3.3 and everything works good.
I did start the bot .. opend my android app.. looked at the map .. switched different screens etc.. Everything works fine.
The difference is that i am using the indego 400 s+ but that should be nearly the same.
Could you please follow the instruction for uninstall / install again .. even if you are doing this already. AND restart your iobroker ?
The theory i have is, due to the 504 errors ( gateway timeout ) .. i think there might be some network issues between your iobroker and the bosch service, that could have many reasons. ( bad internet connection, bad routing from your ips, etc )
Timeout could mean that maybe some requests are running to long and pile up. So that at the end .. too many connections are open.
When i debug this at my machines, this does not happen, but i don t get 504s.
I just changed every request, so that each request can only run once before it can be started again. So the request needs to be completed or failed before it can be started again.
Could you please try Version 0.9.15. I have not had the time to test it for long, but the tests i made worked good.

Could you please try Version 0.9.15. I have not had the time to test it for long, but the tests i made worked good.

Thank you so much!
Did a complete uninstall of the adapter and rebooted the iobroker host, before installing 0.9.15.
No errors so far.

I'll stress test it a bit with extensive use of the app in parallel.

Got errors after some time with 0.9.15.
I have problems with the android app at the same time ("Indego offline" in the app).
So I think it's a problem with the server at  Bosch....
The frequency of the reconnects of the adapter seems right now. Not getting 429 and the adapter isn't force closed by iobroker.

OK good. So the 429s are fixed with the latest changes.
The 429 ( too many requests ) and the 504 (gateway timeout) are errors i can not reproduce nor did anyone else report this issues.
So there does not seem to be a general problem with the servers or the adapter, otherwise everyone else would experience this from time to time.
I still have the hunge that something is off with your setup / network or connection to the bot.
From my side i can do much about the 504 errors.
Was your bot moving during the errors?
So each request ended in a 504?
Just out of curiosity.
Could you try to restart your bot in the following sequence:

switch off the bot ( hardware switch )
pull the bot out of the charging station ( still switched off )
unplug the charing station for 10 minutes
(close the android app / or if you want to try everything, uninstall / reinstall )
plug in the charging station AFTER 10 minutes :)
now - outside of the station -> switch on the bot and then
move the bot to the charging contacts in the station
proceed with startup ( enter pin etc )

Maybe the 504 errors are timeouts because your bot does not respond in time ( there is a mobile network connection / no wifi etc ) inside your bot. Perhaps it has a bad connection. I had some connection problems with my bot, even when it showed a good connection in the display. After doing this routine on 2 days .. it helped. This was a tip from the hotline.

OK good. So the 429s are fixed with the latest changes.
Was your bot moving during the errors?

Yes, it was operating normally and keeps updating the iobroker objects like state text.
So the adapter and my scripts work with 0.9.15!
There are juste these errors in the logs, but the adapter works fine otherwise.

Just out of curiosity.
Could you try to restart your bot in the following sequence:

I'll test it later today and will keep you updated.
Thank you - even more because it seems like its not a common problem!

Just out of curiosity.
Could you try to restart your bot in the following sequence:

switch off the bot ( hardware switch )
pull the bot out of the charging station ( still switched off )
unplug the charing station for 10 minutes
(close the android app / or if you want to try everything, uninstall / reinstall )
plug in the charging station AFTER 10 minutes :)
now - outside of the station -> switch on the bot and then
move the bot to the charging contacts in the station
proceed with startup ( enter pin etc )

Did that with uninstalling the android app.
Unfortunately no change in behaviour. I'll ask Bosch if they or their GSM-provider are having trouble.

Yes,,, but still. Odd that it does not happen on other machines. At least not that massive.
Intersting that your Map errors disapeared after the reset of the bot. I guess it s a mix of server and a connection to your bot.
Or additionaly something with your network connection. ( using proxy, running ioBroker on a vm ? )
I just updated the version to 0.9.16 for you. It s just a slight change.
Before, on each error triggered a reconnect. Now i am ignoring the 504 for operationData ( still showing up in the logs, but not triggering a new connect )
I want to see if this results in a diferent behavior.
Could you please uninstall / reinstall?
What is shown in your datapoints under operatingData?

I let him do a full charge and downgrade the firmware (need to test some others things as well with old firmware).

No errors so far with old firmware 25116. So it seems like there are even more problems with the new one ;)
No errors were thrown by the adapter after software update to FW 26116:

The states during that timeframe (including unknown ones):

OK that looks good. So it was the new firmware. Hmm ..
But when there are other users complaining ( that are not using the adapter .. as this is nothing official, nor do i know if they are too happy about it ) .. i hope they will change the firmware and fix the problems.
Thanks for testing. If you like you can keep me updated on this issue.

Thanks for testing. If you like you can keep me updated on this issue.

Thank you for taking the time! I hope it was worth it just a little bit at least to improve error handling in case of server errors.


fi: nbodykit and dependencies

Adds nbodykit, which is a python-mpi package that itself depends on many python and python-mpi packages. The build systems for many of these packages are very old, but ultimately most just required small tweaks to get to a satisfactory state.
The exception is pfft-python, which downloads a patched fork of pfft during installation. The patched fork itself bundles FFTW; it's pretty deeply integrated and breaking pfft and FFTW out into proper Spack dependencies did not seem easy. The installation seems to go smoothly, though, so I opted to just let pfft-python do this.
I tested the Python packages by importing them, and running pytest on them where applicable.

Superseded by #36.


Custom insertRow modal not working when using react-autosuggest as input

Hello Allen and other fellow developers, im trying to make a custom modal for inserting rows which its actually only and input with autosuggest (using the react-autosuggest module) and it works when put it outside of the modal, the problem is that i need it to be inside of the insert row modal, in fact i need it to be the only thing on the modal other than the confirm button.
i've been trying to get it right using your example and documentation but it just doesn't recognize the custom modal and brings me the default body.
i was wondering if i could get any help on it.
Thank you so much.

HI @sebasvelasco353 sorry for lately reply, how you custom the input or whole the model body or event modal? there are some example you can check :)
Or maybe you can share your code, thanks


[Feature] Create code of conduct

Feature description
Add a CODE_OF_CONDUCT.md file to the project

Hey 👋 @eduardconstantin ,
Please assign this issue to be my name


Move etcd code to its own package in cmd

Move etcd code to its own package in cmd.

I think it's a good idea to move all commands under ./cmd, but separate packages for options is not necessary. We also talked about this with @joelegasse.

There are .DS_Store files being included in this branch that need to be removed. I agree with @gyuho, the options package seems strange. I would rather see a single PR that just moves things around without splitting or changing the underlying packages too much.

Ok, I'll do it.


Bump poetry on CI to >=1.2

https://github.com/ImperialCollegeLondon/FINESSE_GUI/blob/8bb449123d8f4328760ef0a11891eb1dd55a48ea/.github/workflows/ci.yml#L41
Given that there is a new upstream version of poetry and that it seemingly introduces breaking changes, we should probably be using the latest version on our CI.

It looks like the template is still using poetry v1.1 too, so I've opened a PR: https://github.com/ImperialCollegeLondon/poetry_template_2/pull/21


How to do inference?

There doesn't appear to be any examples of how to do inference with this model.
I'm pretty confused, in training you guys feed in the entire label along with the image then use the label for the loss. What am I supposed to feed in if I have a new image?
Additionally, do I just use gpt2-large tokenizer to decode the model outputs?

hi, the inference code is already integrated in the training process.
if you wanna test new images, you can load the best-training model and only do the inference
->do I just use gpt2-large tokenizer to decode the model outputs?
yes, we use GPT-2 tokenizer to decode

Hey thanks! I did manage to find this function which seems to use model.beam_search But I'm still digging through code to figure out how to drop in a new image loader. Is there an example anywhere of using a custom set of images on disk for inference? After I've trained the model, it would be nice to actually use it :)


W3 validation fails with "Element link is missing required attribute property" error

W3 Validator throws an error on the fallback stylesheet:

Error: Element link is missing required attribute property.
<link href="icons.fallback.d9c16476.css" rel="stylesheet">

Code:
<!DOCTYPE html>
<html>
<head>
    <title>Icons</title>
</head>
<body>
    (...)
    <noscript>
        <link href="icons.fallback.d9c16476.css" rel="stylesheet">
    </noscript>
</body>
</html>

Adding property="stylesheet" attribute makes the validator happy.
Somewhat related discussion on Stack Overflow.

Hmm. So this happens when you place the fallback link in the body I guess? Maybe we can add a note to the readme to recommend that, if you place the link in body. (our demos have it in head where this attribute is not necessary)

Since this issue is non-standard and somewhat pedantic, leaving it documented here is good enough.


[Location Registry] Performance Optimisation, response messages and HTTP verbs

What were you trying to achieve?
Test out endpoints for location registry service.
What are the expected results?

Good usage of HTTP verbs
Descriptive response messages
Performance Optimisation

What are the received results?

Usage of HTTP verbs: The Update endpoint could use a more appropriate HTTP verb. That is, we should consider switching from POST to PUT.
Response messages: The response messages for both register andupdateare not very descriptive, could we be able to add more details showcasing what has been registered/updated? And a success field which can be Boolean (true or false)?
Performance: Registration of a new Location takes a very long time.

What are the steps to reproduce the issue?

Git clone this repo and then cd AirQo-api/src/location-registry
Then build the docker image: (docker build -t us.gcr.io/airqo-250220/airqo-location-registry-api .).
Then run the image: ( docker run -d -p 6000:8080 us.gcr.io/airqo-250220/airqo-location-registry-api)
Then test the endpoints using any REST client tool available to you using the chosen port (6000)

base url: http://localhost:6000/api/v1/location_registry
partial url: /register
partial url: /update
More info in the api documentation.

In what environment did you encounter the issue?
REST client tools

no longer valid due to recent changes.


Thank you on behalf of CoverGo

Thank you a lot for the gflows project, @jbrunton!
CoverGo uses it for GitHub-centric CI and it works like a charm.
Let me know if I can buy a coffee for you.
Sorry for the offtopic.

Glad you're finding it useful, @andrey-covergo! Don't worry about the coffee, but lmk if you have any further suggestions!


Get ints and datetimes from .xls files

When using xls file as source, cell containing int value (e.g. year 2018) are returned as float (2018.0), datetime value are also returned as float.
At this time, only boolean cell (XL_CELL_BOOLEAN) value deserve a special processing.
This patch extend processing to :

float to int for number cell (XL_CELL_NUMBER) when no decimals
datetime for date cell (XL_CELL_DATE)

A new xls file data/special/table-with-ints-floats-dates.xls has been added to test conversion

@akariv
Could you please review?
It looks great for me. Related to the very old issue - https://github.com/frictionlessdata/implementations/issues/23

@roll  lgtm!


Support for composite foreign keys?

In the docs, it states that knex has support for composite foreign keys. However, when trying it, I was met with an error (yes, I am using MS SQL, but it seems irrelevant):
TypeError: Cannot read property 'sql' of undefined
    at TableCompiler_MSSQL.TableCompiler.alterTableForCreate (knex\lib\schema\tablecompiler.js:254:45)
    at TableCompiler_MSSQL.TableCompiler.create (knex\lib\schema\tablecompiler.js:93:10)
    at TableCompiler_MSSQL.TableCompiler.toSQL (knex\lib\schema\tablecompiler.js:73:20)
    at TableBuilder.toSQL (knex\lib\schema\tablebuilder.js:72:42)
    at SchemaCompiler_MSSQL.createTable (knex\lib\schema\compiler.js:62:23)
    at SchemaCompiler_MSSQL.toSQL (knex\lib\schema\compiler.js:51:26)
    at SchemaBuilder.toSQL (knex\lib\schema\builder.js:57:43)

Using the syntax the docs referenced:
k.schema.createTable('account', t => {
  t.foreign(['bank', 'office']).references(['office.bank', 'office.office'])
  t.integer('bank')
  t.integer('office')
})

I started to dig into the code, and found no relevant tests for it in postgres, and the code only makes reference to the argument in singular, which leads me to think this is a documented feature, but not an implemented one. To make it work, I ended up writing it raw:
  await k.raw('ALTER TABLE account ' +
    'ADD CONSTRAINT account_office_foreign ' +
    'FOREIGN KEY (bank, office) REFERENCES office(bank, office)')

Which works, but it would be nice to write it alongside the column definitions.

I get this exact same error when running the integration tests.
  1) mssql | mssql Schema createTable rejects setting foreign key where tableName is not typeof === string:
     TypeError: Cannot read property 'sql' of undefined
      at TableCompiler_MSSQL.TableCompiler.alterTableForCreate (lib/schema/tablecompiler.js:9:16687)
      at TableCompiler_MSSQL.TableCompiler.create (lib/schema/tablecompiler.js:9:5173)
      at TableCompiler_MSSQL.TableCompiler.toSQL (lib/schema/tablecompiler.js:9:4314)
      at TableBuilder.toSQL (lib/schema/tablebuilder.js:9:4031)
      at SchemaCompiler_MSSQL.createTable (lib/schema/compiler.js:9:2758)
      at SchemaCompiler_MSSQL.toSQL (lib/schema/compiler.js:9:2279)
      at SchemaBuilder.toSQL (lib/schema/builder.js:9:3010)
      at lib/runner.js:9:3009
  From previous event:
      at Runner.run (lib/runner.js:9:2625)
      at SchemaBuilder.Target.then (lib/interface.js:9:1687)

I observe this error when using both SQL Server 2008 and SQL Server 2017 for Linux (via Docker).

It's interesting that in the case of the test, the error occurs in a test which specifically asserts that this error occurs.  I think it's a bug in the test that the exception is not properly caught.

That specific test seems to test something else, which is the inTable method, which specifies where the table should be joined to. This brought me light into the inTable method however, I didn't see it before! Apparently this does work correctly:
k.schema.createTable('account', t => {
  t.foreign(['bank', 'office']).references(['bank', 'office']).inTable('office')
  t.integer('bank')
  t.integer('office')
})

So the issue becomes:

The references(['table.column1', 'table.column2']) form doesn't work, but references(['column1', 'column2']).inTable('table') does. If we choose to not allow the first form, we should document it properly.
The tests are lacking in this aspect. I didn't see a single example of a composite foreign key and assumed it didn't actually work.
That error is really weird looking, can we prevent it or make it more informative?


Rule out the plugin causes intermittent failures: more logging

To help narrow down causes for https://github.com/scala/scala-jenkins-infra/issues/26


Start without debugging hangs after process exits

VSCode Version: Code 1.17.0 (be377c0, 2017-10-04T23:33:45.303Z)
OS Version: Linux x64 4.8.0-59-generic
I have this launch configuration. It was created by one of the standard templates, I just changed the final arg to ${file}:
{
  "type": "node",
  "request": "launch",
  "name": "Current Mocha test file",
  "program": "${workspaceFolder}/node_modules/mocha/bin/_mocha",
  "args": ["-u", "tdd", "--timeout", "999999", "--colors", "${file}"],
  "internalConsoleOptions": "openOnSessionStart"
}

I select this from the popup list in the debug panel, then hit F5 - my tests run through without issues. I can see the debugger control toolbar pop up and then go away when the tests are done.
When I use Ctrl-F5 to run my tests without debugging, everything still runs through without any issues, the debug console shows the same output from my tests but not the lines starting "debugger listening" and "debugger attached" - so it looks like all is good. However, the debugger control toolbar doesn't go away when the tests are done and I have to stop the process manually by clicking the stop button in that toolbar.
I tried running the command from the console:
node node_modules/mocha/bin/_mocha -u tdd --timeout 999999 --colors myfile.test.js

It runs through and ends without any trouble. I would like code to do that, too...

This might be a long running bug of node not terminating. Though assigning to @weinand to asses this further
@oliversturm as a workaround you can try using protocol: inspector in your launch.json or upgrading to the latest version of node.

@isidorn I guess you know what you're talking about, but node seems to terminate just fine when run from the command line or with the debugger attached. I'm on node v8.4.0 at the moment.

I updated to node v8.7.0 but the problem remains the same.

I noticed that on the vscode side sometimes I am not getting a 'terminated' or 'exit' event from the adapter when running 'start without debugging'. For example for this javascript sample (sorry for the ugly sample)
@oliversturm if you have some minimalistic sample for which you can always reproduce this it would be great. Since it is not always happening, at least to me
function fib(n) { if ((n === 1) || (n === 0)) { return 1; } else { return fib(n - 1) + fib(n - 2); } }

var t = {
	hello: "adsdaddssss",
	second: true,
	third: 5
} // asdlkjasdlkjaslkd jasdjlasdj
// console.log('daslasdljklakdsjkljasdkjlasdkjldalkjdaskljdaskljdaskljasddlaskjdlaskjdalskjdaslkjasdljaslkdjalsjlasdjldasjldkjasldjaslkdjaslkdjlsajdlkasjdlasjdlkasjldjasldjasldjasljldaslasdljklakdsjkljasdkjlasdkjldalkjdaskljdaskljdaskljasddlaskjdlaskjdalskjdaslkjasdljaslkdjalsjlasdjldasjldkjasldjaslkdjaslkdjlsajdlkasjdlasjdlkasjldjasldjasldjasljldaslasdljklakdsjkljasdkjlasdkjldalkjdaskljdaskljdaskljasddlaskjdlaskjdalskjdaslkjasdljaslkdjalsjlasdjldasjldkjasldjaslkdjaslkdjlsajdlkasjdlasjdlkasjldjasldjasldjasljldaslasdljklakdsjkljasdkjlasdkjldalkjdaskljdaskljdaskljasddlaskjdlaskjdalskjdaslkjasdljaslkdjalsjlasdjldasjldkjasldjaslkdjaslkdjlsajdlkasjdlasjdlkasjldjasldjasldjasljldaslasdljklakdsjkljasdkjlasdkjldalkjdaskljdaskljdaskljasddlaskjdlaskjdalskjdaslkjasdljaslkdjalsjlasdjldasjldkjasldjaslkdjaslkdjlsajdlkasjdlasjdlkasjldjasldjasldjasljl')

"C:\Users\User";
console.log('/Users/isidor/Development/w/standup/fib.js');
console.log('server/test.ts');
var t = 'C:\\Users\\User';
var x1 = 9; x1 = 10;
// console.log(msg);
// https://www.google.com
// sdadsasad sdadas dsadasdsa ddassd
var text = "some text";
var t = {
	hello: "asdads"
}
var o = {};
o['hello world'] = 123;
debugger;
var t = 5 / 0;
var m = [];

console.log(['hello', 'world']);
setTimeout(function () {
}, 0);
myLongString
var m = 2 / 0;
var t = "{\nisidor}"; var m = 5; console.log('hello');
let longString = new Array(10001).join('oooooo ');
longString += '<END>';
$isidor

if (true) {
	let longString = 5;
	console.log(longString);
}

// console.log(longString);1

console.log({ foo: "bar" });

var myObject = {
	a: 1,
	b: 2,
	c: 3
}

setInterval(() => {
	// console.log('asdadsads')
}, 500);

var myMap = new Map();
var $isidor = 5;

var keyString = "a string",
	keyObj = {},
	keyFunc = function () { };

// setting the values
myMap.set(keyString, "value associated with 'a string'");
myMap.set(keyObj, "value associated with keyObj");
myMap.set(keyFunc, "value associated with keyFunc");

for (var i = 0; i < 100000; i++) {
	myMap.set(i, "this is entry with: " + i);
}

var text = "    lots     of   whitespace   "

var aLargeArray = []
for (var i = 0; i < 5000; i++) {
	aLargeArray.push(i);
}

var myLongString = "This is a very very very long string that is from the december VSCode release nicely wrapped in the debug console for better visibility.This is a very very very long string that is from the december VSCode release nicely wrapped in the debug console for better visibility.Thi.";
console.log(myLongString)
var c = console;
var t8 = 5;

@isidorn The problem seems easy to reproduce for me. My launch configuration is in the initial post, and I just tried whether it depends on the test - it doesn't. I can repro with this test code:
const { describe, it } = require('mocha');
const assert = require('chai').assert;

describe('barg', function() {
  it('argh', function() {
    assert(42 === 6 * 7, 'oh noes');
  });
});

I'm using mocha 4.0.1 and chai 4.2.1. Node still 8.7.0. I guess that's it... can you see the same?

Yeah this is easy to repro - until recently, we actually couldn't detect when the Node process was finished, when a debugger was attached. When that was hooked up, I didn't fix this noDebug case.


Xcode build error: include of non-modular header inside framework module

Steps to Reproduce
Create a plugin package as per the documentation using Swift in iOS and build it for ios:
flutter create --template=plugin -i swift hello
cd hello/example
flutter build ios --no-codesign

Logs
...
Failed to build iOS app
Error output from Xcode build:
↳
    ** BUILD FAILED **

Xcode's output:
↳
    === BUILD TARGET hello OF PROJECT Pods WITH CONFIGURATION Release ===
    <module-includes>:1:9: note: in file included from <module-includes>:1:
    #import "Headers/hello-umbrella.h"
            ^
    xxx/hello/example/ios/Pods/Target Support Files/hello/hello-umbrella.h:13:9: error: include of non-modular header inside framework module 'hello': 'xxx/hello/ios/Classes/HelloPlugin.h'
    #import "HelloPlugin.h"
            ^
    <unknown>:0: error: could not build Objective-C module 'hello'

Recently upgraded to flutter beta. I've created plugin packages before in the alpha channel without problems.
Flutter Doctor
[✓] Flutter (Channel beta, v0.1.5, on Mac OS X 10.13.3 17D102, locale en-NL)
• Flutter version 0.1.5 at /Users/kozw/Source/flutter
• Framework revision 3ea4d06340 (11 days ago), 2018-02-22 11:12:39 -0800
• Engine revision ead227f118
• Dart version 2.0.0-dev.28.0.flutter-0b4f01f759
[✓] Android toolchain - develop for Android devices (Android SDK 27.0.3)
• Android SDK at /Users/kozw/Library/Android/sdk
• Android NDK at /Users/kozw/Library/Android/sdk/ndk-bundle
• Platform android-27, build-tools 27.0.3
• Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment (build 1.8.0_152-release-915-b08)
[✓] iOS toolchain - develop for iOS devices (Xcode 9.2)
• Xcode at /Applications/Xcode.app/Contents/Developer
• Xcode 9.2, Build version 9C40b
• ios-deploy 1.9.2
• CocoaPods version 1.4.0
[✓] Android Studio (version 3.0)
• Android Studio at /Applications/Android Studio.app/Contents
• Java version OpenJDK Runtime Environment (build 1.8.0_152-release-915-b08)
[✓] VS Code (version 1.20.1)
• VS Code at /Applications/Visual Studio Code.app/Contents
• Dart Code extension version 2.9.2
[✓] Connected devices (1 available)
• iPhone 8 • D5FC98FA-D6B0-41D2-B834-B7FFFEB1B506 • ios • iOS 11.2 (simulator)

I get this too when trying to make a Swift plugin inside the flutter project. I'm using the command:
flutter create --template=plugin -i swift -a kotlin cbdb

and I get the same error sans file path:
xxx/cbdb/example/ios/Pods/Target Support Files/cbdb/cbdb-umbrella.h:13:9: Include of non-modular header inside framework module 'cbdb': 'xxxx/cbdb/ios/Classes/CbdbPlugin.h'

Pretty much renders us making plugins in Swift as a no go. This is critical for us as the Flutter community for plugins is young, and we need plugin builders. I personally am trying to make a wrapper for Couchbase-Lite 2.0.0B, so that we have a proper JSON store with search capability for 100% offline apps in iOS and Android.
I don't get this error in ObjC tests but we're not going near ObjC.
Thanks for the great work so far BTW - Flutter is awesome!

Looks like the created Podfile is the culprit. When I remove Podfile.lock and replace Podfile in the example/ios folder with a version I took from the flutter alpha branch, the build succeeds.
Attached the Podfile from the alpha branch if you want to try it out for yourself (make sure you remove Podfile.lock first). I'm no Xcode build expert, I'm unaware of any unwanted side effects. Use at your own risk.
Podfile.zip

@kozw You saved my life!!! Thanks

@kozw Thanks for working around it. As an iOS dev (Swift) - I'm now doing Android dev to prove the Flutter PoC for our company. Seriously, not easy to wrap CouchBase Lite :) i'm 70% there.

#15763

Looks like the issue is resolved in the master branch. See also #15763

Scratch that, error still exists in Flutter beta v.0.2.8

Looks like the real solution can be found here => #16049

Duplicate of #16049


Redis doesn't return results when using query endpoint

I've added documents to the Redis search using the upsert API, and the documents were added successfully. Still, when I want to use the query API, the results returned are always empty, I have tried different files with different types (PDF, DOCX), but nothing returned in the response.

When you uploaded your document with Upsert, What did the debug look like? Here is an example
mimetype: application/pdf
file.file: <tempfile.SpooledTemporaryFile object at 0x12f164f60>
file:  <starlette.datastructures.UploadFile object at 0x12f1640f0>
extracted_text:
INFO:     <IP_ADDRESS>:55035 - "POST /upsert-file HTTP/1.1" 200 OK

When you query, What does the response look like? Here is an example of one that worked, and two that didn't.
Worked
INFO:     <IP_ADDRESS>:54955 - "POST /query HTTP/1.1" 200 OK
Error
INFO:     <IP_ADDRESS>:55152 - "POST /query HTTP/1.1" 422 Unprocessable Entity
Error: 1 validation error for DocumentChunkWithScore
metadata -> source
  value is not a valid enumeration member; permitted: 'email', 'file', 'chat' (type=type_error.enum; enum_values=[<Source.email: 'email'>, <Source.file: 'file'>, <Source.chat: 'chat'>])
INFO:     <IP_ADDRESS>:55154 - "POST /query HTTP/1.1" 200 OK

The response for uploading the documents was:
mimetype: application/pdf
file.file: <tempfile.SpooledTemporaryFile object at 0x7ff6eae56560>
file:  <starlette.datastructures.UploadFile object at 0x7ff6eae566e0>
extracted_text:
INFO:     <IP_ADDRESS>:56796 - "POST /upsert-file HTTP/1.1" 200 OK

The query returned 200 but still the response is empty

@zainabalthafeeri1 You can validate that the documents are indeed indexed in Redis? (use redis-cli to inspect). Any other details you can provide?

I'm getting the same error here.
Uploaded documents via the /upsert endpoint.
the source fields of all my documents is the string "file".
any idea?

@zainabalthafeeri1 You can validate that the documents are indeed indexed in Redis? (use redis-cli to inspect). Any other details you can provide?

Yes, the documents were indexed in Redis with its respective embedding

the sample docs have "source_id", remove that and delete all documents with the delete endpoint

I just encountered the same issue where Redis doesn't return any results when I provide the "document_id" in the filter:

/upsert endpoints call is successful
Results are returned with this command: curl -H "Authorization: Bearer {token}" -H "Content-Type: application/json" -X POST -d '{"queries": [{"query": "query with documents id"}]}' http://<IP_ADDRESS>:8000/query
An empty response is returned with this command: curl -H "Authorization: Bearer {token}" -H "Content-Type: application/json" -X POST -d '{"queries": [{"query": "query with documents id", "filter":{"document_id": "290d130f-4517-4ff7-a88b-f648df1c44da"}}]}' http://<IP_ADDRESS>:8000/query. The 'document_id' is the same as the one in the response from the successful and /upsert call.
When switching to Milvus, the above commands return results.

I an running the redis and milvus using the docker-compose in chatgpt-retrieval-plugin/examples/docker.

I just encountered the same issue where Redis doesn't return any results when I provide the "document_id" in the filter:

/upsert endpoints call is successful
Results are returned with this command: curl -H "Authorization: Bearer {token}" -H "Content-Type: application/json" -X POST -d '{"queries": [{"query": "query with documents id"}]}' http://<IP_ADDRESS>:8000/query
An empty response is returned with this command: curl -H "Authorization: Bearer {token}" -H "Content-Type: application/json" -X POST -d '{"queries": [{"query": "query with documents id", "filter":{"document_id": "290d130f-4517-4ff7-a88b-f648df1c44da"}}]}' http://<IP_ADDRESS>:8000/query. The 'document_id' is the same as the one in the response from the successful and /upsert call.
When switching to Milvus, the above commands return results.

I an running the redis and milvus using the docker-compose in chatgpt-retrieval-plugin/examples/docker.

Hey @ccqpein this looks like something fixable, can you open a separate issue please and share the same info (and more info about reproducing it with sample data)? cc @isafulf

@ccqpein I think I have a fix for your issue.

@ccqpein I think I have a fix for your issue.

Hi @tylerhutcherson , I just pull the latest main branch and run the test. Unfortunately, this time redis doesn't return the result whether I give it document_id or not. This is the step and the commands I use:
# in chatgpt-retrieval-plugin/examples/docker/redis
docker-compose up -d

# back to chatgpt-retrieval-plugin
# DATASTORE="redis"
# BEARER_TOKEN and API token are in env also
poetry run start

# in other shell
# upsert the read me of chatgpt-retrieval-plugin
curl \
      -H "Authorization: Bearer {token}" \
      -X POST \
      -F 'file=@README.md;type=text/markdown' \
      http://<IP_ADDRESS>:8000/upsert-file

# return 200 and the document id

curl -H "Authorization: Bearer {token}"\
      -H "Content-Type: application/json"\
      -X POST \
      -d '{"queries": [{"query": "start up"}]}' http://<IP_ADDRESS>:8000/query

# return empty results.
# and in terminal that poetry is running, it give a log "Failed to query, error: 'MilvusDataStore' object has no attribute 'col'"

@ccqpein I think I have a fix for your issue.

Hi @tylerhutcherson , I just pull the latest main branch and run the test. Unfortunately, this time redis doesn't return the result whether I give it document_id or not. This is the step and the commands I use:
# in chatgpt-retrieval-plugin/examples/docker/redis
docker-compose up -d

# back to chatgpt-retrieval-plugin
# DATASTORE="redis"
# BEARER_TOKEN and API token are in env also
poetry run start

# in other shell
# upsert the read me of chatgpt-retrieval-plugin
curl \
      -H "Authorization: Bearer {token}" \
      -X POST \
      -F 'file=@README.md;type=text/markdown' \
      http://<IP_ADDRESS>:8000/upsert-file

# return 200 and the document id

curl -H "Authorization: Bearer {token}"\
      -H "Content-Type: application/json"\
      -X POST \
      -d '{"queries": [{"query": "start up"}]}' http://<IP_ADDRESS>:8000/query

# return empty results.
# and in terminal that poetry is running, it give a log "Failed to query, error: 'MilvusDataStore' object has no attribute 'col'"

Everything is fine when I use milvus. Thanks!

This particular issue looks like something different. upsert-file is not working properly for Redis. Opening a new issue

my PR is open which should address this and other things!


Updated footer section

Change :

It also responsive !

@akshattiwarii sure you can work on it also
make sure to follow the contributing.md properly especially the commit part it will help you get the co author badge
also join the discord server for more communication

make sure to complete it in the next 2 days

Hello, I want to resolve this issue please assign to me.


Softmax kernel

I think https://github.com/ROCm/triton/pull/633#issuecomment-2332406892 and https://github.com/ROCm/triton/pull/633#issuecomment-2332425921 is also applicable to this PR.

I think #633 (comment) and #633 (comment) is also applicable to this PR.

Fixed


Suggestion: partial params

Hello!
Sometimes its nice to have an option to pass custom params into a partial. Like that:
<partial name="block" :add-link="true">
The principle I see is the same as lightweight VMs in v-for elements.
This would also require (probably) alternative syntax for partials:
partials: {
    block: {
        props: {addLink: null},
        template: '
123
'
    }
}

You could say "use components then", but the point is I don't want most of components' features here. Just new data scope.

You should just use a component! :smile:
Look at #2221

What is the problem with using components? :D
Just exchange partials with components and go on to real problems :)
components: {
    myBlock: {
        props: { addLink: null },
        template: '
123
'
    }
}

<component is="myBlock" :add-link="true">

@Nirazul The case is exactly like with v-for instances. Actually, I want all parent component data to be inherited. Its like using now gone inherit: true. Overall its very expensive in terms of performance. Using lighweight VM will lower the cost.

@Nirazul If you take a look at v-for, you can see it works exactly like I suggested for partials to work.
It creates new data scope, adding 1-2 additional props (key/item) to the scope, inherits parent's data.
I Could even use v-for for the case, repeating the partial with specialy created data objects.

@Nirazul I mean, like that
{
    prop1: {value: true},
    prop2: {value: false}
}

    <partial name="block"></partial>

    <partial name="block"></partial>

I see, what you mean although I haven't been in the situation yet, where I wished this existed.
It definitely would help to encapsulate some partials and improve reusability.

@Nirazul in this case, your partial has to be part of the component, right?

@oliverzy I'm not sure on what part you are replying to, but if you refer to my first comment: You can also use dynamic components with <component :is="myComponentNameString">. They should be registered globally beforehand, of course.

@Nirazul okay, I understand, thanks

Partials are something I always want to move away from: it's a composition mechanism very similar to string templates which I personally never use in my code.
I think if you are not concerned with performance, you should absolutely just use components instead. If performance is absolutely critical, then just replicate the blocks without using partials - because partials although being cheaper than components, still also has its own perf cost.


Split Grid components

:tada: This PR is included in version 1.5.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Chat - "New Messages" badge is displayed when navigating to the link of a message

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 9.0.64-0
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught on HybridApp, is this reproducible on New Expensify Standalone?: Y
Issue reported by: Applause - Internal Team
*
Action Performed:

Open the Expensify app
Open any chat with long message history
Scroll up until the bottom of the chat stop being visible and long tap on a message
Select "Copy Link
Navigate to a different chat and paste the message´s link
Tap on the just sent link
Verify you are redirected to the correct chat and message and that the "New Messages" badge is not displayed

Expected Result:
When navigating to a message´s link, the user should land on the correct chat, the message should be highlighted and the "New Messages" badge should not be displayed
Actual Result:
When the user navigates to the link of a message on a chat with long history, the "New Messages" badge can be seen for some time when landing on the message
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Standalone
[ ] Android: HybridApp
[ ] Android: mWeb Chrome
[ ] iOS: Standalone
[ ] iOS: HybridApp
[ ] iOS: mWeb Safari
[ ] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos

Add any screenshot/video evidence

https://github.com/user-attachments/assets/d19de2b7-e4a1-4be2-a823-5abb0a37ab15
View all open jobs on GitHub

dupe https://github.com/Expensify/App/issues/52782


Possibly missing an import in Getting Started 101

At https://github.com/scailable/sclbl-tutorials/blob/master/sclbl-101-getting-started/README.md the line sp.upload(lm, fv, docs=docs) uses a variable sp. I could not see any reference to this, but assume somewhere there should be import sclblpy as sp.

Hi,
Thanks for the message; I just added the import statement.


Cannot find package "github.com/db-journey/migrate/v2"

$> GOPATH=/tmp go get github.com/db-journey/journey
package github.com/db-journey/migrate/v2: cannot find package "github.com/db-journey/migrate/v2" in any of:
	/usr/local/go/src/github.com/db-journey/migrate/v2 (from $GOROOT)
	/tmp/src/github.com/db-journey/migrate/v2 (from $GOPATH)
package github.com/db-journey/migrate/v2/driver: cannot find package "github.com/db-journey/migrate/v2/driver" in any of:
	/usr/local/go/src/github.com/db-journey/migrate/v2/driver (from $GOROOT)
	/tmp/src/github.com/db-journey/migrate/v2/driver (from $GOPATH)
package github.com/db-journey/migrate/v2/file: cannot find package "github.com/db-journey/migrate/v2/file" in any of:
	/usr/local/go/src/github.com/db-journey/migrate/v2/file (from $GOROOT)
	/tmp/src/github.com/db-journey/migrate/v2/file (from $GOPATH)

$> go version
go version go1.13.4 linux/amd64

I guess on top of "naming things" and "cache invalidation", we could add "go modules". Sorry for the mess, I tried everything I could, and ended up editing go.mod by hand, and now it seems to work. Could you please retry? thanks

No pb :)
We are almost there, no more issue with go modules, but now this:
/tmp > GOPATH=/tmp go get github.com/db-journey/journey
# github.com/db-journey/journey/commands
src/github.com/db-journey/journey/commands/commands.go:26:17: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag in array or slice literal:
	cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver)
src/github.com/db-journey/journey/commands/commands.go:30:4: unknown field 'EnvVar' in struct literal of type cli.StringFlag
src/github.com/db-journey/journey/commands/commands.go:32:17: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag in array or slice literal:
	cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver)
src/github.com/db-journey/journey/commands/commands.go:36:4: unknown field 'EnvVar' in struct literal of type cli.StringFlag
src/github.com/db-journey/journey/commands/cronjobs_commands.go:14:3: cannot use startCommand (type cli.Command) as type *cli.Command in array or slice literal
src/github.com/db-journey/journey/commands/migrate_commands.go:30:3: cannot use createCommand (type cli.Command) as type *cli.Command in array or slice literal
src/github.com/db-journey/journey/commands/migrate_commands.go:31:3: cannot use upCommand (type cli.Command) as type *cli.Command in array or slice literal
src/github.com/db-journey/journey/commands/migrate_commands.go:32:3: cannot use downCommand (type cli.Command) as type *cli.Command in array or slice literal
src/github.com/db-journey/journey/commands/migrate_commands.go:33:3: cannot use resetCommand (type cli.Command) as type *cli.Command in array or slice literal
src/github.com/db-journey/journey/commands/migrate_commands.go:34:3: cannot use redoCommand (type cli.Command) as type *cli.Command in array or slice literal
src/github.com/db-journey/journey/commands/migrate_commands.go:34:3: too many errors

That's the joy of modules again. You need to use GO111MODULE=on to fetch the right version of urfave/cli. They unfortunately started a new V2 directly in master instead of ./v2 as described here https://blog.golang.org/v2-go-modules.
See https://github.com/urfave/cli/issues/459

You are right, all good now!
Thx.

They unfortunately started a new V2 directly in master instead of ./v2 as described here https://blog.golang.org/v2-go-modules.
See urfave/cli#459

👀 I may be working on exactly that problem => https://github.com/urfave/cli/issues/952


Bug Reporting 🐞

Reporting Bugs ❗

If you find any bugs or issues while using this project, report the bug here and we will fix it as soon as possible. Active found bugs will be listed down below and will be removed if fixed.

Known Bugs 🐛

Quit undefined error when typing q to quit a game after playing against bots.

Quit undefined error when typing q to quit a game after playing against bots.

Quit undefined error when typing q to quit a game after playing against bots.

@ShawnDriscoll thanks for the report! I will fix this.

@ShawnDriscoll Were you using the exe or the python version when you found this error?

The EXE. I didn't check if the .PY has the same issue.

@ShawnDriscoll This has now been fixed!


Don't make a specific mobile app

Just a recommendation. Just make it a responsive web design.
Assuming that https://github.com/OpenSprites/OpenSprites/tree/master/public_html/app is a "mobile app"

For your information, we are (trying to) making the website design as responsive as possible.
.. Though I forgot why some of the team wants to create an app.. It was decided months ago and no progress.

I think it was primarily arinerron's idea.

@Firedrake969: We already decided to make an app months before Arinerron joined. And during your inactivity, too. :P

oh


Main table sorting

The main table appears to only sort in one direction with the largest number on top. When enrichment are calculated, one typically want the smallest number on top. It was be handy for the sort icon to be an ascending/descending toggle, for this reason - more intuitive than scrolling all the way to the bottom.

when an annotation list is uploaded and gene symbols don't match PANDA's internal list (e.g. noncoding genes like RP5-862P8.2), there is a pop-up showing the list of IDs that didn't map. After the pop-up box is gone, there appears to be no way to get this "unmapped IDs" file... Perhaps you could add a little [U] icon to the left of the [X] icon (for delete annotation collection) that is a link to this file.


Naming scheme

The naming scheme currently implemented is insecure and broken for several other reasons as well. The main problem is that you have a deterministic top level name, and only inside that a possibly non-deterministic second name.
On a multi-user system, the top level directory (e.g. /tmp/foo) might already exist, but belong to some other user. Depending on your situation, this could mean your fixture run interfering with that of someone else, or your fixture run failing because someone else owns a directory of that name which is not world-writable, or perhaps the owner has set strange attributes for that directory (like sticky group bit or some ACLs) which make files created inside that directory writable to him. If the fixture executes any code in such a temporary directory, this could lead to one user gaining the privileges of another.
All of the above is theoretic for me, since in practice things fail in the common use case: if the named top level directory does not exist already, and you want to copy data from somewhere else and didn't name the subdirectory explicitely, then wrench.copyDirRecursive is used instead of mkdirp, and will fail since the top level directory under which it should create the child directory does itself not exist.
All things considered, I believe it would be better if the top level directory were non-deterministic, perhaps with a prefix to indicate where it belongs. In my use case I've ended up using temp. It isn't using mkdtemp either, but it puts so much entropy in the name that a clash is very unlikely. And its track() method to clean up any left-over stuff automatically at exit is nice, too. Perhaps you want to build on that?

Ah, you are correct. The current implementation does not do any valid checking of folder existence/ownership before working within it. At the time I wrote this library, I was not aware of the security implications and we should fix them.
I don't exactly understand your deterministic/non-deterministic terminology. Are you implying that /tmp or its variant isn't guaranteed to exist on an OS? I believe tmp is covering that case fine:
https://github.com/raszi/node-tmp/blob/v0.0.23/lib/tmp.js#L43-L54
If not, can you provide more examples? (e.g. example paths and where the problem lies in those paths).

I use “deterministic” to mean “non-random” and vice versa. And I wasn't concerned about the existence of /tmp itself, but the directory below that. The one you have to pass to the FixtureDir, which I called “top-level”, as opposed to the one you may pass to mkdir which I called “subdirectory”.
Let's have an example. If I simply want any directory for my code, which is for the tests of module “foo”, then I'd do
var fixtureDir = new FixtureDir("foo-tests");
fixtureDir.mkdir(callback);

and get something like /tmp/foo-tests/16952H0dMU15N4tWk. Now the problem here is /tmp/foo-tests. It might exist, be owned by someone else, and have strange properties. It could be a symlink, or be set up so that filles dropped in there become writable by someone else.
I propose you don't include a trailing slash with your prefix. And you leave directory creation to tmp.dir. That way you end up with some /tmp/foo-tests-16952H0dMU15N4tWk which is guaranteed to be newly created, since mkdir fails for existing dirs. It also resides directly in /tmp, which in turn should not have any evil attributes set if the system is set up in a regular way. It is created with access only for the current user, so anything you do inside is safe as well. If the user passes a folder name to mkdir, then you can set up a folder of that name inside the randomly named folder you just created.
When I first compared the tmp and temp modules, I hadn't realized that tmp did such randomization as well, nor that it's already used in your code. Now the two seem pretty much equivalent, and sticking with tmp is fine. It looks as if tmp would by default delete the directories it created at exit, just like temp can do with the track() setting, so switching to tmp.dir is enough to give you that feature as well.

Ah, I understand now. That sounds like a good proposition. I am going to take a look at getting those changes in place.

After starting to dig into the code and refresh my memory further, I am realizing that this was initially built to make debugging tests easy as well. Tests should be cleaning up their /tmp folders every time which wouldn't lead to this as a problem. Can you elaborate on the use case more?

Let's assume that project “foo” contains the following test case:
var fs = require("fs"),
    path = require("path"),
    child_process = require("child_process"),
    FixtureDir = require("fixture-dir"),
    fixtureDir = new FixtureDir("foo-tests");

fixtureDir.mkdir({}, function(err, dir) {
  if (err) throw err;
  // simulate unpacking some tarball or similar
  fs.writeFile(path.join(dir.path, "configure"), "echo All is well");
  setTimeout(configure.bind(null, dir), 5000);
});

function configure(dir) {
  child_process.spawn("/bin/sh", ["./configure"], {
    cwd: dir.path,
    stdio: "inherit"
  }).on("close", function(code) {
    dir.destroy();
  });
}

Scenario 1: unintentional clash
Alice and Bob work on the same machine on that project. At some time, Alice runs the above test. Some time later, Bob does the same. He receives an error message:
Error: EACCES, mkdir '/tmp/foo-tests/1961196q1rkw'
    at Error (native)

That's because /tmp/foo-tests exists, belongs to Alice and isn't writable by anyone but Alice.
Scenario 2: malicious attack
Bob and Eve work on the same machine. Bob is the sole developer of the project containing the test above. The machien is set up to clear the /tmp directory every night. Eve runs the following shell script before Bob starts his work:
mkdir /tmp/foo-tests # new directory belongs to Eve, so she can do stuff
chmod a+w /tmp/foo-tests # but Bob can write, so he won't notice anything
setfacl -d -m other::rwx /tmp/foo-tests/ # newly created files become world-writable!
while ! [ -e /tmp/foo-tests/*/configure ]; do :; done # wait till Bob runs his test
sleep 1 # wait till the write is complete so we are the last to write that file
echo "echo You got hacked" > /tmp/foo-tests/*/configure # overwrite

Now Bob will unintentionally execute the code Eve wrote to his file. It might be installing Eve's ssh public key in Bob's ~/.ssh/authorized_keys file, so Eve can log in as Bob and do anything from there.

I think we are conflating 2 issues in a single issue thread.

Naming (e.g. 2 users are attempting to talk to the same fixtured folder for tests)
Permissions (e.g. /tmp/foo-tests should be owned by user running tests and only writable by them as well)

I agree that we should patch permissions (scenario 2). With respect to naming (scenario 1), my concern is that we lose the ease of debugging since as a developer I need to jump folders every time I run tests. To reiterate my question, is scenario 1 hypothetical or is it actually occurring for you?

I am also wondering if scenario 1 is rare enough that we could have an option for developer's to provide the randomized string themself. For example:
// Generate fixture dir with random name (e.g. `foo-tests-1pfn788c684`)
new FixtureDir('foo-tests-' + (Math.random() * 1e16).toString(36))

Those two issues have a single solution. The only portable way I can think of to ensure sane permissions is by creating the directory from scratch.
Can you elaborate on why you'd have to “jump folders”? Is it in some shell, or in the test suite? Is it while the test is running, or afterwards, or if it failed? Is it to repeatedly execute the same commands, so that a * in the path could take care of the unknown name? The subdir is expected to go away after each test, so you shouldn't cd to that if the tests completes successfully, right?
So far, scenario 1 isn't an issue for me, since I haven't started using fixture-dir on a large scale due to the issues here, and since I'm mostly using my computer just for myself. Things would become different if I'd set up some CI which runs as a different user.
More importantly, though, I don't feel comfortable publishing software when I know that users building that software might be vulnerable due to an issue I know about. So issue 2 is my main concern.

Ah, concurrent CI tests under the same user is definitely a valid user case and scenario for 1 (e.g, Jenkins where we don't have VMs).
Even while random names obscures the problem we should still have programmatic enforcement of permission checking (if a folder already exists). For example, a hacker could fill all possible folder names or somehow predict our random seed.
By "jump folders", I mean that I sometime have a terminal open in the fixtured directory and inspect its contents (e.g. I am in /tmp/foo-tests and look at cd bar/ to see what the git status is).

For concurrent tests, sometimes CI will provide known different variables to prevent collisions. In those scenarios then a user could provide that random string from process.env whereas our random generation might not be guaranteed to avoid collisions =/

Alright, whipped up a branch that enforces users/permissions. Does this look good to you?
https://github.com/twolfson/fixture-dir/compare/ddf041d3eb64783a7b191e22c735cc4c62901785

ddf041d3eb64783a7b191e22c735cc4c62901785 looks safer at least.
It won't address scenario 1 above, since if the folder has the wrong owner, you error out. But there is little you can do, if you insist on your naming convention, since you can hardly delete a folder owned by somone else. Personally I'd prefer desiredMode = 0700 over desiredMode = 0777 & (~process.umask()) since that way the code I'm testing can do unsafe stuff in there, like executing code in world-writable files, and I won't have to worry about it. I guess things would break e.g. on some FAT-based file system, where you don't have ownership information.
I guess I'll stick to tmp.dir or temp.mkdir since I prefer the auto-cleaning and the way scenario 1 won't happen. But that's because I never expect to change to the fixture dir manually, so I don't care what it's named. It's a matter of taste and usage scenario, no longer a security issue.

Can you elaborate on your desiredMode notes further? With respect to directories, read allows for listing files and execute alllows for accessing said files. They aren't attached to the permissions of the files themselves.
With respect to randomizing naming, I am leaning toward making it easier to opt-in to randomization than not. What's your opposition to the randomization snippet I posted earlier?
// Generate fixture dir with random name (e.g. `foo-tests-1pfn788c684`)
new FixtureDir('foo-tests-' + (Math.random() * 1e16).toString(36))

Closing issue due to inactivtity


MiKo_6040 is now aware of leading comments when moving nodes

(resolves #1163)
@CodiumAI-Agent /describe

Title
MiKo_6040 is now aware of leading comments when moving nodes

User description
(resolves #1163)
@CodiumAI-Agent /describe

PR Type
Bug fix, Tests

Description

Fixed a bug where comments preceding multiline call chains were removed during formatting.
Enhanced SyntaxTokenExtensions to correctly handle leading spaces when comments are present.
Updated StringBuilderExtensions to optimize string comparison logic.
Added new tests to ensure comments are preserved in multiline call chains.

Changes walkthrough 📝
Relevant filesEnhancement

StringBuilderExtensions.csAdjust string comparison logic for better performance

MiKo.Analyzer.Shared/Extensions/StringBuilderExtensions.cs
Modified logic for handling string differences in QuickCompare.  Switched conditions for using rented arrays versus direct comparison.

+5/-5

Bug fix

SyntaxTokenExtensions.csImprove handling of leading spaces and comments in syntax tokens

MiKo.Analyzer.Shared/Extensions/SyntaxTokenExtensions.cs
Added logic to handle leading spaces when comments are present.  Introduced Spaces method for consistent space handling.  Refactored methods to improve handling of leading trivia.

+51/-11

Tests

MiKo_6040_MultiLineCallChainsAreOnSamePositionAnalyzerTests.csAdd tests for multiline call chains with comments

MiKo.Analyzer.Tests/Rules/Spacing/MiKo_6040_MultiLineCallChainsAreOnSamePositionAnalyzerTests.cs
Added tests to verify comment preservation in multiline call chains.  Ensured tests cover scenarios with single and multiple comments.

+126/-0

💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information


APIs for non-actor state transactions

Describe the proposal
For this PR, state transactions for non-actor apis are implemented.
Add documentation to
https://github.com/dapr/docs/tree/master/howto/create-stateful-service
https://github.com/dapr/docs/blob/master/reference/api/state_api.md

Documented here: https://docs.dapr.io/reference/api/state_api/#state-transactions


UserSecrets: remove build warning and tweak runtime error

PM's don't like the UserSecrets build warning when using Microsoft.AspNetCore.All. So removing it.
Closes https://github.com/aspnet/Configuration/issues/651
Incidentally closes https://github.com/aspnet/Configuration/issues/548 and closes https://github.com/aspnet/Configuration/issues/547 which were about making improvements to the build warning.

cc @danroth27 @DamianEdwards

:bell:


Allow reinits in sub procs

Before submitting

[x] Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
[x] Did you read the contributor guideline, Pull Request section?
[ ] Did you make sure to update the docs?
[ ] Did you write any new necessary tests?
[ ] If you made a notable change (that affects users), did you update the CHANGELOG?

What does this PR do?
Fixes #981
When using multiprocessing the default behavior of wandb is to return None for multiple calls to init.  Passing reinit=True ensure we always create an experiment.  This could lead to cases where a single experiment has multiple runs in wandb, but it's a much better default than throwing an exception.
PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃

@vanpelt thx for you work, could you please check my comments so we can get it done :]

Hey, any update here?

@vanpelt mind rebasing? looks like there are conflicts

@vanpelt mind allow edit for maintainers?

Finished in #1360


Override equals in Model

We are using Mock library for creating our unit test, and some of the nswag functions has a parameter that is a nswag model, I would like to override the equals of that model so I can successfully mock the function.
Is it possible to do it?

You can override a template (for example Class.liquid): https://github.com/RSuter/NJsonSchema/wiki/Templates
https://github.com/RSuter/NJsonSchema/blob/master/src/NJsonSchema.CodeGeneration.CSharp/Templates/Class.liquid

I've added Class.Body.liquid so you can add additional class body code without replacing the whole class template (and miss future fixes in it).

(will be available in the next release)


Check for actual start in SSE

Closed #1566  in favor of this.
We're just gonna leave LP alone since it's the last transport in the fallback loop and don't actually need to detect successful start up.
That means a lot less churn on the client and sever and in the tests.
You'll see that this change is much smaller.

Added some tests. Ready for review


Banchmarkdotnet

Can you please rerun the performance test with BenchmarkDotNet and share the results?

Done check Readme.md for results.


Generate both Objective-C and Swift docs using Jazzy

Requires https://github.com/realm/jazzy/pull/341. Output can be browsed here: https://static.realm.io/jazzy_demo/Realm/
/cc @tgoyne @bdash @timanglade

I took a quick look at the output and noticed that most of the "Show on GitHub" links have incorrect line numbers. For instance, the documentation for -[RLMObject linkingObjectsOfClass:forProperty:] links to line 381 of RLMObject.h.

That's because the GitHub link is being generated as if these were the docs at v0.96.2. If you replace that with jp-jazzy-objc, it'll lead to the right place: https://github.com/realm/realm-cocoa/blob/jp-jazzy-objc/Realm/RLMObject.h#L381

:+1:


the items dropped by the open crate despwan and need a support block.

not sure if I get the way the block works right, but I guess it's supposed to make the block in front of it like a transparent chest, but in reality it works exactly like a automatic dropper.

It is ment to work like an automatic dropper, it drops items when you insert into it, and it can pickup items if you extract.


Enable the Latest release link on safe_browser repo to use as download link from all websites

To avoid having to update all webites' download link, we should try to enable the Latest release link on github for the safe_browser repo:

https://help.github.com/articles/linking-to-releases/#linking-to-the-latest-release
https://github.community/t5/How-to-use-Git-and-GitHub/Why-last-release-is-not-marked-as-quot-latest-release-quot/td-p/1762

For a quick test, we can publish a dummy release from a fork without the "pre release check box" and see if its getting the "latest release". If it does, might make a case to apply the "pre release checkbox" only for our rc (release candidate) releases which the latest release URL will hopefully not pick.

Resolved by PR's:
https://github.com/maidsafe/dev_website/pull/46
https://github.com/maidsafe/safenetwork.tech/pull/100

ok, as mentioned above it was just about removing the prerelease checkbox, so the links work correctly now.

Closing this as https://github.com/maidsafe/safe_browser/releases/latest is already enabled.


⚠ api.piste.gouv.fr healthcheck has degraded performance

In 99c1824, api.piste.gouv.fr healthcheck (https://api.piste.gouv.fr/cassation/judilibre/v1.0/healthcheck) experienced degraded performance:

HTTP code: 200
Response time: 4743 ms

Resolved: api.piste.gouv.fr healthcheck performance has improved in 5c2de06.


Submit via enter

I actually implemented this during hour hack hours, but forgot to push. So, now you can submit by pressing Enter.

Alrighty, thanks for taking such a close look @mpanne, I'm gonna close this for now because fixing the issue would require more than 10min of my time, so maybe let's revisit this some other time. :)

So simple and so great. 👌
You wanna add a test maybe? If not, fine with me^^
Fun Fact: I can't approve this as it is my PR - consider it approved though. :D


java.io.IOException: No available port for the BitTorrent client!

I'm getting this error, and have a couple of questions:
Here's my code I run for each torrent file I'm sharing:
	Client client = new Client(InetAddress.getLocalHost(),
				SharedTorrent.fromFile(torrentFile, outputParent));

		client.setMaxDownloadRate(50.0);
		client.setMaxUploadRate(50.0);
		client.share();

Do I have to instantiate client for every new torrent file? It seems to me that I should have a single client, that can add or remove torrent files.

If this is the case, then the port ranges, should be similar to other clients, usually, 49152-65535

Yes, a Client in this case deals with a single torrent file.


MEMPTR not marked as volatile

In libmos6502vm/types.h:
#ifndef __HOST_C__
#define MEMPTR(address)       ((uint8_t*)(address))

Because address is not marked as volatile, the compiler is allowed by the C standard to elide (remove) multiple writes to it. This causes the 0xcafe benchmark to not pass on LLVM-MOS, because it gets converted from C to the following LLVM IR:
define dso_local i16 @main() local_unnamed_addr #0 {
entry:
  tail call void @_puts(ptr noundef nonnull @.str) #2
  tail call void @_putc(i8 noundef zeroext 67) #2
  tail call void @_putc(i8 noundef zeroext 65) #2
  tail call void @_putc(i8 noundef zeroext 70) #2
  tail call void @_putc(i8 noundef zeroext 69) #2
  tail call void @_puts(ptr noundef nonnull @.str.1) #2
  tail call void @_putc(i8 noundef zeroext 10) #2
  ret i16 0
}

These serial calls to _putc are then inlined further into writes to memory location CPORT, but as they are not volatile writes, the C compiler may assume it is legal to remove duplicate writes into the same memory address if there are no reads happening from that address. This is why the code then returns 0XCAFE == 0XE :).
The proposed fix is to adjust this to:
#ifndef __HOST_C__
#define MEMPTR(address)       ((volatile uint8_t*)(address))

To clarify, this fixes the 0xcafe and aes256 benchmarks for llvm-mos; I have not tested other compilers, as I don't have all of them set up.

Thanks, @asiekierka! Fixed.

Note that SDCC also uses volatile (at least I've seen it being used in the GBDK toolchain). It's part of the C standard, after all.

(Also, just out of curiosity, what's with all of the custom LDCOPTS like -mllvm -force-precise-rotation-cost? We don't use them in mainline llvm-mos-sdk; if they actually provably help, we'd like to know about this, and if they don't, it might be best to keep as close to llvm-mos-sdk's official setup as possible.
If you want to use our official compiler runtime and such, you might be better off using mos-common-clang and adding your own link script on top of it.)


Using back or forward mouse-buttons always results in uncaught exception

An uncaught exception was reported.
Info:
OS: win32
Arch: ia32
GPMDP Version: 3.0.1
Time: Thu Mar 17 2016 17:23:24 GMT-0600 (Mountain Daylight Time)
Error:
Attempting to call a function in a renderer window that has been closed or released. Function provid
Error: Attempting to call a function in a renderer window that has been closed or released. Function provided here: Object.<anonymous> (C:\Users\username\AppData\Local\GPMDP_3\app-3.0.1\resources\app\build\inject\GPMInject\interface\mouseButtonNavigation.js:8:8.
    at BrowserWindow.callIntoRenderer (C:\Users\username\AppData\Local\GPMDP_3\app-3.0.1\resources\atom.asar\browser\lib\rpc-server.js:201:19)
    at emitTwo (events.js:92:20)
    at BrowserWindow.emit (events.js:172:7)

This issue was created automatically inside the uncaughtException handler

Closing as duplicate of #470


Not exactly an issue, but...

I've done a fair amount of work on this, adding in beat detection, more effects, and tidying up some of the code to make it more flexible for new effects - Here's a quick preview
Should I submit a pull request?

It might not be added, as Scott hasn’t been on in a bit, but I’d absolutely love to see it!

I would love it too ! It seems to be amazing 👍. If you plan to upload your code somewhere, let me know it :)

I've forked it and will start pushing updates as they're ready

I'd love to see the beat detection part.
Also, if we could have a spectrum mode with a logarithmic scale instead of linear  it would be great. (Actually, is it linear as of now ? I can code - it's my job -, but I tried to understand the audio analysis part of the code and oh man audio analysis seems to be a very "niche" thing... lots of math, and I don't really know where to start looking).
At least, what I found is that in a lot of songs, the lower frequency part of the spectrum is always very bright and middle/highs do not often light up as much. Then the second "problem" I see is because of the linear aspect of the spectrum, the majority of instruments (bass, guitars, voice, etc.) only light up a quarter of the band, and the three quarters left lights up with like a cymbal... and everything lights up because cymbal makes a lot of different high frequencies. I'm not sure I'm very clear....
It's the same as looking at a spectrum analyzer with music playing live, and making it linear or logarithmic. A lot of things that we hear as different (making up the melody etc.) are between 20 and... 5000k Hz I'd say. 5k to 12k are very high frequency. It should not take 2/3 of the led strip.

Yeah I've already implemented the logarithmic scaling. I'm thinking about also scaling the brightness depending on the percieved volume of different frequencies, using a mel scale or something similar. I'll keep you all posted on what is going on.

Righto, it's up on my page. Here's a preview

That looks like it will be able to do some nice stuff, are you planning on adding fadecandy support?

Had a look at Fadecandy, and I must say it looks like a very powerful tool. Very visually impressive too! If you could give me a demo or something showing how it can work better than the current library, I'd consider adding it in.
My current order of jobs is:
1 - Add customisable options to the rest of the remaining effects
2 - Add a couple more effects when I come up with something. Open to any suggestions! If you can describe it, I'll take a shot at making it.
3 - Support for multiple strips.
If you

I will upload a video of a fadecandy running the experimental code when I get home tonight. As far as I can tell the main upgrade with the fadecandy would be improved dithering over the esb2866, 5v support without a inverter and higher fps. I was hitting 300 fps with no buffering the other night.

Hello. Take issue with your fork.
Python 2.7 error
Traceback (most recent call last):
  File "C:\Users\Loatan\Desktop\audio-reactive-led-strip-master\python\visualization.py", line 690, in <module>
    gui = GUI()
  File "C:\Users\Loatan\Desktop\audio-reactive-led-strip-master\python\visualization.py", line 345, in __init__
    super().__init__()
TypeError: super() takes at least 1 argument (0 given)

That's a python 2 error, if you run it in python >3.5 it doesn't appear. This new code also requires scipy and pyaudio 5 so make those changes

Would you be interested in making a pull request? These effects look nice

@Hexalyse I've just adjusted melbank.py on my fork to give better scaling as you described. The this fork has already already implemented logarithmic scaling, but it's now much so.
Here's the mel graph before, and after. As you can see, lower frequencies are stretched out much more, and higher noes are more squished together.
I've found this looks much better on the strip, as vocals now appear across the centre, lows at the bottom, and highs exclusively at the top.

@not-matt I already checked your fork a few days ago (didn't have a lot of time with holiday season). You've done an incredible job already !
The only thing I noted was that in the "Energy" type of visualization, most of the "strip" will be full white, and only the borders are colored (It's like it "clips" or "peaks" in an area that is a bit wide). I don't know if it would be possible to have an option to adjust how much and how fast the different color channels "scale down" while going farther and farther from the middle ? I don't have a good enough understanding of how the audio processing part of the code works to get how it's done.
I will try your fork with the new mel graph later ! :) I think this project has a good potential if we (you, mostly) keep adding features.
I was thinking about other effects but I'm too tired to describe them right now. I'll let you work on what you have on your to-do list, and we'll see about it later.

@Hexalyse I'll have a look at it and see if I can figure something out. My actual LED strip is in my university room, while I'm now at home for Christmas, so this might be one for when I can see the strip.
Do keep me posted with visualisation ideas, I'm all out now. Here's my last one: Power, available on my fork.
With regards to that to-do list, I'm not sure if it's worth adding multiple strip support. What do you think? Any other features/ideas? As I've said, I'm all out of ideas now...!

@not-matt here is a short video showing a fadecandy running a big matrix in autocorrelation on the experimental build.

Forgot the video https://youtu.be/uUySLVwkrZE

@charlieali Fadecandy looks really good! I've had a look at their GitHub page too, and it certainly seems like a very powerful tool. However, there are a couple of problems I can see:

Fadecandy runs the effects using C++, whereas this project is entirely done in Python.
All of the current visualisation effects are only designed to work on a 1d strip. It would take a lot of work to rewrite the Visualization class to support 2d or 3d effects.
Fadecandy seems to run in its own sandbox, so I'm not sure how it would be able to get audio input.
It would require purchasing Fadecandy boards

Some things that look very promising about Fadecandy (copied from their GitHub):
"""

A high performance USB protocol
Interpolation between keyframes
Gamma and color correction with per-channel lookup tables
Temporal dithering
"""

Interpolation to boost FPS seems like a pretty neat feature, and the dithering looks very nice too. One thing to consider is whether these features can be implemented into the ESP8266 sketch. If so, that would give some of the key benefits without the need to rewrite a lot of code to switch to this library. I don't know anything about dithering, but the demo looked good. Do you think the colours on the ESP8266 strip look as rich as on the Fadecandy controlled matrix?
One last thing - If you have any suggestions for visualisation effects, do let me know! I would love some ideas from everyone 👍

I  took some video with a better camera to put up later today, but yeah it's pretty powerful. I will try to post one with a single strand so you can see what a 1d scroll looks on it. It's basically a teensy 3.0 with a level inverter and 100 ohm resistors built in. The colors and dithering are much better on the fadecandy than on an esb board. As far as running it the fadecandy runs on a server and you just tell the server what it needs to do and it does it, I don't know exactly how Scott Lawson implemented it in the experimental code, I'm not a developer I just mess around with things. If you ever decide to implement it and need a chip you can send me an Amazon wish list and I'd be glad to gift you one.
As far as new effects, I will try to think of some.

Any chance of a link to this experimental code? I'd be very interested in having a look, with the possibility of adding it in.

https://github.com/scottlawsonbc/audio-reactive-led-strip/tree/experimental?files=1
It's just a branch

Oh that should be no issue to add into this new fork. I'll let you know when it's done!

@charlieali  Drop me an email here containing your email address so I can contact you directly?
<EMAIL_ADDRESS>

Wow, the 2D effect looks reaaaally nice. I might consider purchasing a Fadecandy if it's worth it. But I guess LED matrices cost a lot of money.
BTW, @not-matt I quickly tried your fork with the new effects and mel graph yesterday (I had very little time to test it since I had a friend home, but she really enjoyed it and was hypnotized by the LED strip haha). I noticed that the spectrum looked much better, but I had an issue with the "Energy" mode and even more with the "Scroll" mode : on the original code by scott, I had a pretty wide variety of colors (bass - low frequencies - yielded more redish colored while high pitch yielded blue/white, and volume made it vary too). In your last version - I guess you changed a few things, alongside the mel graph ? - the entire LED strip is pretty much only white. It's like every color channel goes way too high. Same for the energy mode : before I had the center of the strip white, and nice colors on the side of it. Now, even when the volume of the song is not super high, I get almost all the strip taken, and 95% of the LED are pure white, with a very tiny (as in not large) portion of the extremities are colored. And I can see it on the GUI too : all three color channels are clipped, and not just at the center, like almost all the way through the "graph".
I wish I had time to dwelve into the audio processing part of the code so I can understand and debug it myself, but I'm afraid I like some knowledge (and time) here.

@hexalyse I didn't buy it, I made it. it's about $200 in materials with the fadecandy. Drilling the holes for the LEDs was tedious but not too hard. The reason the effect looks the way it does is because the LEDs are technically snaking up and down which provides a nice visual effects.

@Hexalyse I've added a few options to adjust the amount of r,g,b for the effects you mentioned ["Scroll", "Energy", "Spectrum"]. As I've said, I don't have access to an actual strip at present, making it hard to see how colours look, especially when they merge. Please could you have a fiddle around with each effect and give me some values that look good? They're currently all at a default of 1.00, with the sliders ranging from 0.05-1.00. I'll set the values you give as default so it looks nice for everyone, but leave the sliders in anyways just in case someone wants to change them for themselves.

Here are 2 more videos of the fadecandy running a matrix:
Chrome Sparks - Send the Pain On - https://youtu.be/X11aMf1rSbo
deadmau5 - FN Pig - https://youtu.be/TAZmYzFTlA4
@scottlawsonbc if you want to use them as examples for the experimental code go ahead.
"
Bonus video of my christmas tree which has a matrix "star" running this code off a fadecandy, the tree is ran off a teensy with an octows2811 shield running Jinx!

Good lord this looks beautiful. Can't wait to get into programming this!
On 29 December 2017 at 00:04, charlieali<EMAIL_ADDRESS>wrote:

Here are 2 more videos of the fadecandy running a matrix:
Chrome Sparks - Send the Pain On - https://youtu.be/X11aMf1rSbo
deadmau5 - FN Pig - https://youtu.be/TAZmYzFTlA4
@scottlawsonbc https://github.com/scottlawsonbc if you want to use them
as examples for the experimental code go ahead.
"
Bonus video of my christmas tree which has a matrix "star" running this
code off a fadecandy, the tree is ran off a teensy with an octows2811
shield running Jinx!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/scottlawsonbc/audio-reactive-led-strip/issues/97#issuecomment-354377453,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Ae5azNXwQUKsEExUK2gn35WwnMNfQa5hks5tFCyQgaJpZM4Q_v1Y
.

@charlieali I'm not sure I understood what you meant. Are you using the current experimental branch of the audio-reactive-led-strip repo ? And you matrix are just Neopixel LEDs connected in a zig-zag-ing pattern (going up and down, left to right) and the visualization in spectrum mode ?
But what surprises me again is that when I use the visualization mode almost avery LED are light up. Not full bright, so I see it following the music, but I never have a majority of LED being OFF and others following frequencies like in your videos. Did you increase the "FFT bins" number way up ? (I was not sure what it would change exactly)
@not-matt I'll try to clone and test your new settings during the day and get back to you with what looks best. For more precision I don't use a microphone. I use "Audio-router" to route the audio output to my soundcard (I use an external USB Asus soundcard) AND a virtual audio cable that I then set as default recording device, thus getting good audio input in Python.

#@hexalyse yeah it's strings going up and down running the experimental branch. Autocorrelation with 512 fft bins and I believe gnuplot and prism_r as colors.

@hexalyse I get plenty of darkness, don't know why you have the issue you have. Here it is running with 128 fft bins cycling through some of the simpler colors. I also took the blanket I use for diffusion off. https://youtu.be/bjycrtuE3ik

@charlieali Well I try using 30 bins instead of the default 24 (since I use 60 LEDs, 30 on each side, I guessed 30 bins would be the best way to have a spectrum covering all the LEDs ? But I don't really know what these bins correspond to) and yeah, I never get as many black LEDs. Most of them are middle-purple, and going to white.
I guess the best would be to share screenshots of the graph shown in the GUI instead of speaking about LEDs, 'cause this graph shows the exact RGB values sent to the LEDs.
I'll try getting screenshots of all of this when I report about my results.

But I guess there is some logic here : when you get a bright LED for a very "thin" band of frequencies, I, on the opposite, will get a bright LED for this note, but it will be included on a much larger band of frequencies. Since I have less LEDs, I cannot "differentiate" as many bands (bins ?) as you do.
To be more clear : let's say we create a sound with frequencies that will light up one every four LEDs on your setup (and the other 3 are not lit because no frequencies correspond to what they represent)... on mine, all LEDs will be lit up since all these frequencies will fall in a "bin" corresponding to a LED on my strip.
That's the only explanation I can come up with, right now, just by thinking about how it would do that.

Right let me just explain what's causing this issue

@Hexalyse mentioned a while back that it would be good if beats weren't as bright, highs didn't take as much of the strip, and mids were more clearly visible. This was made possible by adjusting the mel graph, to give a more even spread of pitch across the output. For effects such as Wavelength, this looks good, because lots of different parts of the strip light up for different tones. You get more evenly distributed pitch/light. However, as you've described, this doesn't look as good on effects such as Scroll, because all the different frequency bins are responding together. In this effect, they all overlap, giving a mostly white strip.
So that's the problem. Changing the number of FFT bins will not make much of a difference I'm afraid. However, the solution is pretty simple and I'll sort it after adding the Fadecandy support: Custom Y scaling per effect. For Wavelength, putting the y values to the power of a half will make loud bands quieter, and quiet bands louder. The opposite is true for effects where you want more difference between bands (like Scroll or Energy): Putting the Y values to the power of 2 or 3 will make loud bands louder and quiet bands quieter.

@not-matt Thanks for the detailed explanation. I wish I could dwelve into this part of the code and understand it, to make better feedback. Anyway, you seem to be doing a very good job and adding many visualizations !
BTW, since @scottlawsonbc seems to be back and active (on the experimental branch), would it be feasible to merge your work on these new visualizations with configurable parameters, and his work on the new devices etc. ?
Also, you say that this effect comes from the modified mel bank, but talk about a "custom Y scaling". Could it be better/different to (instead/also) allow to dynamically change the mel bank ? Or choose a different one for each effect ?

@not-matt
Thanks for your work - i'm completely new to this and now I'm trying both versions :) The new effects introduced by your branch are very nice, but one thing confuses me a bit. With scotts release the strip behaves in a way that not always every single led lights up - something I really like.
When I try your fork at about 90% almost every led lights up - is there some kind of setting I can change? I would love to have the behavior of scotts original version with your additional effects :)

@relax81 Are you using my fork, or the pull request on this one?

@not-matt
I've used this one
https://github.com/not-matt/audio-reactive-led-strip
The only issue was that the microphone.py caused a problem while starting the visualization.py - so i had to change it back to the original one to make it work. But it looked liek there weren't any big changes made in that particular part of it (?)

Quick fix then, make the following changes.
Melbank.py
Uncomment lines 48, 63
Comment out lines 49, 64
Visualisation.py
Comment out line 888
Add in below it: mel = mel**2.0

Thanks, I'll try it out.

Same too, cannot wait to see what commits are in the future.Really awesome what you're doing.
Feature request:
With the added support of two rgb strips coming up, would be cool to have left and right audio matching left and right rgb strips.
Are reaching the limit of raspberry pi and ESP8266? What would you recommend going forward hardware wise, or what are testing with?

Would anyone with multiple esp8266s be willing to test a little something on my fork?

@Mattallmighty I run the program on my win10 using a hotspot and a couple of ESPs, and have no idea how it would run on a pi. I've added support for a couple things like Fadecandy and something else, but haven't tested those either.
If you're thinking about remote control, sometime in the future I'm planning a web based front end for controlling the lights. I've got a pi running a home server which I plan on using to control the strips 24/7, then I can control them through my phone or something. Still thinking about it though, not too sure.
With regards to LR audio, it would be cool, but definitely not worth the overhaul that it would require! Some things are much easier said than done.

@not-matt
More than happy to test multiple strips, I have 2 at the moment, and have additional 6 on the way in March. More than happy to test extensively. Got your pull at the moment and setup on a single strip and looking great :D
I am using ESPs too, will order some more today. Getting 10 or so NodeMCU v3 as need some spares.

Are you posting any updates in your page or are you doing a pull request on
this one?
On Feb 26, 2018 3:48 PM, "Mattallmighty"<EMAIL_ADDRESS>wrote:

@not-matt https://github.com/not-matt
More than happy to test multiple strips, I have 2 at the moment, and have
additional 6 on the way in March. More than happy to test extensively. Got
your pull at the moment and setup on a single strip and looking great :D
I am using ESPs too, will order some more today. Getting 10 or so NodeMCU
v3 as need some spares.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/scottlawsonbc/audio-reactive-led-strip/issues/97#issuecomment-368644710,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AcIJYN8vtTLOTQlDryCuoM8RD92yXdffks5tYxh0gaJpZM4Q_v1Y
.

I'll upload the latest version once I finish reworking the colours. The gradients currently use linear changes between colours but that doesn't look great. I'm adding in smooth transitions, it shouldn't take too long.
Here's a visual representation
Also a couple of effects are broken because of all the changes going on behind the scenes. This will be fixed so don't open new issues about it. I get the impression you want to get your hands on the latest version so I'll upload it even though a few effects aren't working fully.

I'm not in a hurry, just wanted to know if you plan on continuing this fork
or commit to your own fork since with all you seem to be adding you are
moving pretty far away from Scott's. Pretty far into awesome. I will be
messing around with your Christmas update later once I figure out why my
esb won't talk to my desktop. Will post results.
On Feb 27, 2018 11:47 AM, "not-matt"<EMAIL_ADDRESS>wrote:

I'll upload the latest version once I finish reworking the colours. The
gradients currently use linear changes between colours but that doesn't
look great. I'm adding in smooth transitions, it shouldn't take too long.
Here's a visual representation https://imgur.com/a/5GwFM
Also a couple of effects are broken because of all the changes going on
behind the scenes. This will be fixed so don't open new issues about it. I
get the impression you want to get your hands on the latest version so I'll
upload it even though a few effects aren't working fully.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/scottlawsonbc/audio-reactive-led-strip/issues/97#issuecomment-368944568,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AcIJYDQLr-qhXrMBK7CDWQv-tpupHLrhks5tZDGVgaJpZM4Q_v1Y
.

@charlieali It depends really - If I were promoted to some role where I could push updates to this fork then I would carry on here. This would be ideal since most people follow this fork rather than mine.
On the other hand, I'll probably continue pushing to my fork simply because it is a fork I can actually update. If I had a more able role here then I wouldn't have need to use my fork.

Understandable. Either way you are doing cool things, good job m8
On Feb 27, 2018 12:30 PM, "not-matt"<EMAIL_ADDRESS>wrote:

@charlieali https://github.com/charlieali It depends really - If I were
promoted to some role where I could push updates to this fork then I would
carry on here. This would be ideal since most people follow this fork
rather than mine.
On the other hand, I'll probably continue pushing to my fork simply
because it is a fork I can actually update. If I had a more able role here
then I wouldn't have need to use my fork.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/scottlawsonbc/audio-reactive-led-strip/issues/97#issuecomment-368959453,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AcIJYKAlVT9IsRtMNUzzX0P7oHOs55Kcks5tZDu2gaJpZM4Q_v1Y
.

Ok so I've put up the new update on my fork. Things added/changed include:

Multiple devices. You can have as many as your computer can handle, different types too.
Support for new devices. Fadecandy and Dotstar (both untested, please lmk if you get them to work - have a look through devices.py)
Settings persistence. Program will remember stuff (usually) from where you left it on your last session. Just make sure to exit gracefully using that little cross in the top corner, no hammernig ctrl-c. If stuff starts going wrong or you can't launch the program for weird reasons, try deleting settings.ini in lib
COLOURS are now much more fun and colourful. Have a nosy around the bottom of config.py - you can make your own colours and even colour gradients now, it's really easy. If you make any especially pretty ones please let me know and I'll add them in - the list is a little sparse right now. The gradient graphs do look really weird now, but on the actual strips they are a million times better.
Scroll and Energy are broken but I'll fix them soon. It's because of old code not being able to handle the flexibility of multiple devices and signal processing, they require a few random globals to work which I've been getting rid of. It's just a case of re-writing those effects.

I've probably forgotten a couple of things but that's that for now. As always, let me know of any bugs that I haven't mentioned. I would also love to see some of the setups you guys are running!
 Going that far with such a simple solution and making it so, so much better is really amazing. Thanks a ton, from all of us.

Cool stuff, trying to get fadecandy going now. Installed opc with pip but
still getting importerror no module audioled.opc I will see if I can't
figure it out.
 Going that far with such a simple
solution and making it so, so much better is really amazing. Thanks a ton,
from all of us.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/scottlawsonbc/audio-reactive-led-strip/issues/97#issuecomment-369075464,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AcIJYLgYUyvZFC0VR886gF7WjVhaDHiYks5tZJv8gaJpZM4Q_v1Y
.

Hey guys,
Decided to move my fork over to a seperate repo so that I can control pushes to master. Scott has put in tremendous effort developing this program and laying the framework for what I want to build, but the thing is this: I want to be able to create this freely, and part of allowing that process to flow is having the control of the repo to do so.
I would hugely appreciate the support in development that you guys give. I want to be able to more effectively open and close issues on my project alongside everyone, but right now that flexibility isn't quite there, considering Scott isn't on here very often these days.
If anyone is confused or needs a hand moving over (which they will, considering the bugs that have been found already), you have a choice. Remain on this repo by simply rolling back to an older pull request, most likely my one from Christmas, or come with me to my repo where I will be answering questions and moving the project forward.
With that, I consider this issue closed. If there is a commit to pull, I would reckon the Christmas update would be it, but that probably won't happen for a while. After all, I would say this is not exactly an issue... 😉 Please continue discussion on this topic over here
Thanks again for your support everyone, and keep those bugs coming!
Matthew