[Bug]: Can't create any database : file cannot contain any of these characters

Contact Details
No response
What happened?

I tryed to create a new simple database
I set the name "test" and the current folder as source

The following message appear :

I don't have any log related

What platform were you using?
Desktop
Version of the plugin
2.6.2
Relevant log output
No response
Are you using the latest version of the plugin in your Obsidian vault?

[X] I have verified that I am on the latest version

Are you check if there is a similar issue?

[X] I have verified that there are not similar issues

with the same names and target path I can not reproduce this issue, maybe fix with another fix?

Archive.zip
Try this version to discard if its a possible fixed bug

Nop, doesn't work.
I think it's maybe an error that occur only on windows, where path contains \ instead of / ? But I think that Obsidian fix that in their different function that includes path...

I will test it on Windows to see if I can reproduce it

Obsidian manage that relative paths internally. I could test it today on windows and its working cool

I retry today, and no success to create a database using the command.
No error in log, just this message about incorrect naming.

Could you share your active plugins? maybe exists some kind of incompatibility.

I disabled all my plugins (keeping only db folder & dataview) and It continues to do that.
Interestingly, it doesn't do that in my other instance of Obsidian.
At the moment, the only difference between my two vault :

The place (one on I:\, other in D:\Documents\Github\Pingouin)
Syncing system : I use Obsidian Sync on the main vault, and none in the second (it's my vault test)

SYSTEM INFO:
	Obsidian version: v0.16.3
	Installer version: v0.14.15
	Operating system: Windows 10 Education 10.0.22622
	Login status: logged in
	Catalyst license: insider
	Insider build toggle: on
	Live preview: on
	Legacy editor: off
	Base theme: light
	Community theme: Minimal
	Snippets enabled: 44
	Restricted mode: off
	Plugins installed: 88
	Plugins enabled: 2
		1: DB Folder v2.6.7
		2: Dataview v0.5.46

RECOMMENDATIONS:
	Custom theme and snippets: for cosmetic issues, please first try updating your theme and disabling your snippets. If still not fixed, please try to make the issue happen in the Sandbox Vault or disable community theme and snippets.
	Community plugins: for bugs, please first try updating all your plugins to latest. If still not fixed, please try to make the issue happen in the Sandbox Vault or disable community plugins.

I don't do anything strange with the create of the file, just the use of the Obsidian API
  /**
   * Add new file inside TFolder
   * @param file_path
   * @param filename
   * @param content
   */
  async create_markdown_file(targetFolder: TFolder, filename: string, localSettings: LocalSettings, databasefields?: RowDatabaseFields): Promise<TFile> {
    LOGGER.debug(`=> create_markdown_file. name:${targetFolder.path}/${filename})`);
    const created_note = await app.fileManager.createNewMarkdownFile(
      targetFolder,
      filename ?? "Untitled"
    );
    let content = databasefields ? parseFrontmatterFieldsToString(databasefields, localSettings).concat("\n").concat(parseInlineFieldsToString(databasefields)) : "";

    // Custom content by source
    switch (localSettings.source_data) {
      case SourceDataTypes.TAG:
        content = content.concat(`#${localSettings.source_form_result}\n`);
        break;
      default:
    }

If you enable the plugin logs on debug mode (developer general settings section), we can see the exact generated path
LOGGER.debug(=> create_markdown_file. name:${targetFolder.path}/${filename}));
With that path you can reproduce it with the API directly on the console of Obsidian
app.fileManager.createNewMarkdownFile(targetFolder,filename ?? "Untitled")
Maybe Obsidian API its not prepared for some Windows disk as I://

closed as wont fix


Provide helpful error message for not providing C to heatmap

hvplot=0.8.1
related to https://github.com/holoviz/holoviews/issues/5444
I was trying to display a heatmap and I received an error message. I got it working and then during my refactor got the error message again. The error message was not indicative of the error in any way and I needed to debug into the hvplot code to understand the issue.
I.e. I wasted a lot of time.
The issue is that the script
import plotly.express as px
import hvplot.pandas
import numpy as np
df=px.data.tips()
df["count"]=1
df.hvplot.heatmap(x="smoker", y="day", c="count")

gives the error
ValueError: Dimensions must be defined as a tuple, string, dictionary or Dimension instance, found a NoneType type.

The cause is the c instead C. Some kind of validation and helpful error message would be really appreciated here to make hvplot easy to use.

Just noting that C is an optional parameter of the heatmap function.


Add icon for “Automated readability index”

I suggest adding an icon for the exercise "Automated readability index".
I am attaching one of the possible options for the icon.

Hey, thanks for finding this. Exercise icons live in the exercism/website-icons repo.


Set Permissions in Built Packages

We should have permissions set correctly before the worlds are added to the tarball. We should add this to the workflow instructions for building worlds.

Fixed in 0.2.0


Update next-meeting-topics.md

Neat visual transaction editor someone made: https://www.bitcointech.wiki/editor
Channel factories without a soft fork: https://github.com/JohnLaw2/ln-factory-optimized/blob/main/factoryoptimized13.pdf
"In addition, a version of the FFO protocol, called the Fully-Factory-Optimized-Watchtower-Free (FFO-WF) protocol, is presented that supports watchtower-freedom and one-shot receives for casual users [Law22a]. No change to the underlying Bitcoin protocol is required for any of these protocols"
A different channel factory proposal from that one: https://eprint.iacr.org/2018/918.pdf
"We obtain better performance results by a factor of more than 3000 times with respect to DMC Factories. From a cryptographic point of view, our solution requires for multi-signatures a non-interactive aggregate signature scheme. Maxwell et al. [10] recently proposed a scheme for Schnorr Multi-signatures with applications to Bitcoin. This scheme is however, interactive, not non-interactive, as we require. For this reason, we apply the BNN [1] non-interactive scheme for our Lightning Factory, instead of Schnorr"
Very exciting times!

Did you... did you just comment on my PR instead of making your own? 😂

Did you... did you just comment on my PR instead of making your own? 😂

I tried to make my own. It said I couldn't and I was unable to recover what I wrote. It was very sad.

Conflicts/old - if this is still something to talk about, please rebase!


remove codeless install

@aaustin please leave on branch until ready for release. Now that we've added handling for race conditions that presumably were not an issue before this, I need to go test that removing the load call doesn't re-break any functionality.

Seems like it'd be better to find a working solution than just drop this. Ideally, calling getInstance vs getTestInstance would actually get you different instances. This is something that probably should have been done all along.

I'm not sure what solution is workable. Since this occurs at load time, we need some way to know whether getInstance or getTestInstance is going to be invoked in application:didFinishLaunchingWithOptions:. One hacky, partial solution I came up with was to not fire this if branch_key is a dictionary and test is a key that's present. Then codeless installs would have to refrain from including a test key.
For now, until we can decide on functionality, we need to remove this, as it's already generated confusion for 2 partners.


License Names missing from Feed2Catalog

The Feed2Catalog job should also include any License names (and associated fields) when it creates a catalog entry.

These are added now. When a catalog is recreated they should be there.


External dependencies and Karma

IMPORTANT: This repository's issues are reserved for feature requests and bug reports. Do not submit support requests here.
Steps to reproduce and a minimal demo of the problem
npm install -S openlayers
Added mapping according to https://github.com/mgechev/angular2-seed/wiki/Add-external-dependency
What steps should we try in your demo to see the problem?
Current behavior
npm run test fails with 404, cannot find '/base/node_modules/openlayers'
Expected/desired behavior
Other information
Any suggestions or a more detailed example (except for angular-jwt) on how to add 3rd party libraries?
Thank you!

For support requests you can use gitter or take a look at project's the wiki page in order to get more complete idea on how the it works.


How to use ssh

Hi ,thanks for the great tool ,I want to use ssh ,I found scp ,but did not found ssh ,thanks

Hi, could you provide error message when run ssh? Term48 has ssh command by default.

The ssh binary is located inside the Term48 bundle at app/native/root/bin/ssh. The path to this is set in the main executable, so you should be able to run ssh by just typing 'ssh'. If you have reset your $PATH in your shell in .profile or some other configuration file, then you might not be able to find it anymore. In this case, just add the correct path to your $PATH, or don't clobber the default $PATH by using PATH=/my/custom/stuff/:$PATH or equivalent.
The scp you found is probably the one that ships with bb10 out of the box. Term48 also ships a scp that will work, in the same directory as ssh.

HI thanks for replay ,I still not find ssh ,can you give me the Absolute Path of ssh

The ssh binary is at /accounts/1000/appdata/ca.mordak.Term48.gYABgJC2m1SvBDfQMWQBsIohLnw/app/native/root/bin/ssh
Your path there is wrong - it should contain $HOME/../app/native/root/bin. The warning at the top there about no controlling tty is also unusual. Can you tell me what device you have, and what builds of BB10 and Term48 you have installed? If you have a Playbook, then Term48 on that device does not ship with ssh. Old versions of the playbook-dev-tools repo did build on the playbook, so you can build it yourself if you are feeling adventurous, but the playbook has not been really supported for years.

my Q10 is good ,playbook not working ,Thanks.good job


Redundant line

https://3ds.guide/get-started-(new-3ds-usa)
"For all versions you can Cart Update to a higher version in the same column to follow its instructions instead."
The New 3DS guide only has two versions in the same column.  And there is never a reason to cart update from 9.0-10.7 to 11.0-11.2.  So this line should be removed.
EUR version has the same issue.
Technically the JP guide is okay since you can cart update from 8.1, but people reading the guide who don't know any better might think it is useful to cart update from 9.0-10.7 to 11.0-11.2, which is a bad idea, so you may want to take out the line and stick it inside the 8.1 "no browser" table entry instead

Eh, makes it easier for me to maintain in the future as new versions are added. I'll leave it; nobody has mentioned this as as a point of confusion before.


Support for yup-based schema validation

E.g. through another addon in the monorepo, like @ember-headless-form/yup maybe?

what all would be needed for yup support?
name-wise, it sounds great, and with Changesets, managing releases for multiple things in a monorepo is super easy 🎉


Strptime Undefined Reference Error

I get the following error when trying to use the strptime function:
.\obj\test_strptime.o:test_strptime.f90:(.text.startup+0x5d): undefined reference to `__mod_datetime_MOD_strptime'
.\obj\test_strptime.o:test_strptime.f90:(.text.startup+0xdf): undefined reference to `__mod_datetime_MOD_strptime'
collect2.exe: error: ld returned 1 exit status

I'm trying to use the public strptime function like follows:
program teststrptime

use datetime_module

type(datetime)  :: date1,date2
type(timedelta) :: timediff

! Example times in "YYYYMMDD hhmmss" format
character(len=15) :: str1 = "20130512 091519"
character(len=15) :: str2 = "20131116 120418"

date1 = strptime(str1,"%Y%m%d %H%M%S")
date2 = strptime(str2,"%Y%m%d %H%M%S")

timediff = date2-date1

!write(*,*)timediff
!write(*,*)timediff%total_seconds()

end program teststrptime

I am using gfortran GCC 6.3.0 from MinGW on Windows. I have the datetime-fortran library in a lib folder and the mod files in a include folder. I'm compiling like so:
gfortran -O2 -fbacktrace -c test_strptime.f90 -o .\obj\test_strptime.o -L./lib -ldatetime -I./include/ -I./obj -J./obj
gfortran -O2 -fbacktrace -o teststrptime .\obj\test_strptime.o

Using date2num instead of using strptime does not cause an error during compiling.
Any suggestions would be appreciated. I'm still fairly new at linking libraries for Fortran development.

Hi Doug,
Not sure what's going on here. I can compile and link the example without issues on my system (Ubuntu).
This is a linker error so your compile step works. It seems like you have a libdatetime.a (otherwise the linker would complain about it) but it can't find the strptime method in there. Just for a sanity check, what is the output of nm libdatetime.a | grep __mod_datetime_MOD_strptime for you? You should get something like<PHONE_NUMBER>000346 T __mod_datetime_MOD_strptime is libdatetime.a was created correctly.
Unfortunately I have zero experience with MinGW, so I don't know if this could be related to it.

Hey, thanks for the quick reply. Running the sanity command check you mentioned gives:
0000000000002290 T __mod_datetime_MOD_strptime

I think the issue is related to using WSL to build / compile datetime-fortran, but then switching back over to using MinGW to link the library to my project. Unfortunately we're using MinGW and Windows across the project.

Hi,
is there an easy solution to get datetime-fortran working using mingw in windows ?
thanks

Hi @VinniDoddu,
Unfortunately, I haven't learned anything new since the start of this issue, and still have zero experience with MinGW. From reviewing the issue again, my suspicion is that on MinGW the linker is not able to find strptime. This should be in the C standard library inside time.h. I don't know if on MinGW this function is not provided, or perhaps defined in some other place that is not in default include and link path.

@scivision just added a fix for this which is now in master.
@dcdenu4 @VinniDoddu Can you try it and see if it works?

Closing. If the issue persists, please reopen.

@scivision just added a fix for this which is now in master.
@dcdenu4 VinniDoddu Can you try it and see if it works?

Hey, sorry I missed this. I'm no longer on the project that pertained to this issue and don't have access to the codebase... Thanks for the work and reaching out!


FileNotFoundError: [Errno 2] No such file or directory: '/home/jr/uwazi-docker/data/mongo/WiredTiger.turtle.set'

After cloning the code,
docker-compose run -e IS_FIRST_RUN=true --rm uwazi
ends with the following error:
Status: Downloaded newer image for mongo:3.4
Creating uwazi-docker_mongo_1         ... done
Creating uwazi-docker_elasticsearch_1 ... done
Building uwazi
Traceback (most recent call last):
  File "bin/docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 71, in main
  File "compose/cli/main.py", line 127, in perform_command
  File "compose/cli/main.py", line 845, in run
  File "compose/cli/main.py", line 1297, in run_one_off_container
  File "compose/service.py", line 316, in create_container
  File "compose/service.py", line 352, in ensure_image_exists
  File "compose/service.py", line 1067, in build
  File "site-packages/docker/api/build.py", line 154, in build
  File "site-packages/docker/utils/build.py", line 31, in tar
  File "site-packages/docker/utils/build.py", line 79, in create_archive
  File "tarfile.py", line 1803, in gettarinfo
FileNotFoundError: [Errno 2] No such file or directory: '/home/jr/uwazi-docker/data/mongo/WiredTiger.turtle.set'
[4393] Failed to execute script docker-compose
jr@erwin:~/uwazi-docker$ ls data/mongo/WiredTiger.
WiredTiger.lock    WiredTiger.turtle  WiredTiger.wt

Thanks for the report.
If I did not commit or anwser this in 48 hours, can ping me back with a new comment.

Just to give a fast feedback for now:
Here a full log. The main difference is that I had to run docker-compose run -e IS_FIRST_RUN=true --rm uwazi twice (probably because the elastic search was not ready for connections fast enough).
Also the message "WARNING: Image for service uwazi was built because it did not already exist. To rebuild this image you must use docker-compose build or docker-compose up --build." maybe could means some additional difference.
# fititnt at bravo in ~/tmp [20:42:53]
$ git clone https://github.com/fititnt/uwazi-docker.git
Cloning into 'uwazi-docker'...
remote: Enumerating objects: 145, done.
remote: Total 145 (delta 0), reused 0 (delta 0), pack-reused 145
Receiving objects: 100% (145/145), 48.99 KiB | 0 bytes/s, done.
Resolving deltas: 100% (78/78), done.
Checking connectivity... done.

# fititnt at bravo in ~/tmp [20:43:43]
$ cd uwazi-docker

# fititnt at bravo in ~/tmp/uwazi-docker on git:master o [20:43:47]
$ docker-compose run -e IS_FIRST_RUN=true --rm uwazi
Creating network "uwazi-docker_default" with the default driver
Creating uwazi-docker_elasticsearch_1 ... done
Creating uwazi-docker_mongo_1         ... done
Building uwazi
Step 1/9 : FROM node:6-slim
 ---> e7745d9e5093
Step 2/9 : LABEL maintainer="Emerson Rocha<EMAIL_ADDRESS> ---> Using cache
 ---> 8d315a326fe4
Step 3/9 : RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y   bzip2   dh-autoreconf   git   libpng-dev
 ---> Using cache
 ---> 82ab5966ecb0
Step 4/9 : RUN git clone -b 1.1 --single-branch --depth=1 https://github.com/huridocs/uwazi.git /home/node/uwazi/   && chown node:node -R /home/node/uwazi/   && cd /home/node/uwazi/   && yarn install   && yarn production-build
 ---> Using cache
 ---> 964f21c84b91
Step 5/9 : COPY --chown=node:node ./scripts/patch/uwazi/database/reindex_elastic.js /home/node/uwazi/database/reindex_elastic.js
 ---> Using cache
 ---> 6a4614ebfbca
Step 6/9 : RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5   && echo "deb http://repo.mongodb.org/apt/debian jessie/mongodb-org/3.6 main" | tee /etc/apt/sources.list.d/mongodb-org-3.6.list   && apt-get update   && apt-get install -y mongodb-org-shell mongodb-org-tools
 ---> Using cache
 ---> 492e62752d45
Step 7/9 : WORKDIR /home/node/uwazi/
 ---> Using cache
 ---> d4ad8f33205f
Step 8/9 : COPY --chown=node:node docker-entrypoint.sh /docker-entrypoint.sh
 ---> b943b7ba41b1
Step 9/9 : ENTRYPOINT ["/docker-entrypoint.sh"]
 ---> Running in ef77676901a3
Removing intermediate container ef77676901a3
 ---> 2b6495a358c8
Successfully built 2b6495a358c8
Successfully tagged uwazi-docker_uwazi:latest
WARNING: Image for service uwazi was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
uwazi-docker: IS_FIRST_RUN: true
uwazi-docker: Enviroment variable IS_FIRST_RUN is true. Assuming need to install database from blank state

uwazi-docker: Deleting mongo uwazi_development MongoDB database
MongoDB shell version v3.6.4
connecting to: mongodb://mongo:27017/uwazi_development
MongoDB server version: 3.4.14
WARNING: shell and server versions do not match
{ "ok" : 1 }

uwazi-docker: Importing /home/node/uwazi/blank_state/uwazi_development to mongo uwazi_development MongoDB database
2018-10-12T23:44:40.642+0000	the --db and --collection args should only be used when restoring from a BSON file. Other uses are deprecated and will not exist in the future; use --nsInclude instead
2018-10-12T23:44:40.685+0000	building a list of collections to restore from /home/node/uwazi/blank_state/uwazi_development dir
2018-10-12T23:44:40.726+0000	reading metadata for uwazi_development.translations from /home/node/uwazi/blank_state/uwazi_development/translations.metadata.json
2018-10-12T23:44:40.731+0000	reading metadata for uwazi_development.users from /home/node/uwazi/blank_state/uwazi_development/users.metadata.json
2018-10-12T23:44:40.732+0000	reading metadata for uwazi_development.sessions from /home/node/uwazi/blank_state/uwazi_development/sessions.metadata.json
2018-10-12T23:44:40.732+0000	reading metadata for uwazi_development.settings from /home/node/uwazi/blank_state/uwazi_development/settings.metadata.json
2018-10-12T23:44:40.997+0000	restoring uwazi_development.translations from /home/node/uwazi/blank_state/uwazi_development/translations.bson
2018-10-12T23:44:41.270+0000	restoring uwazi_development.settings from /home/node/uwazi/blank_state/uwazi_development/settings.bson
2018-10-12T23:44:41.443+0000	restoring uwazi_development.sessions from /home/node/uwazi/blank_state/uwazi_development/sessions.bson
2018-10-12T23:44:41.632+0000	restoring uwazi_development.users from /home/node/uwazi/blank_state/uwazi_development/users.bson
2018-10-12T23:44:41.633+0000	restoring indexes for collection uwazi_development.sessions from metadata
2018-10-12T23:44:41.634+0000	no indexes to restore
2018-10-12T23:44:41.634+0000	finished restoring uwazi_development.settings (1 document)
2018-10-12T23:44:41.634+0000	no indexes to restore
2018-10-12T23:44:41.634+0000	finished restoring uwazi_development.translations (1 document)
2018-10-12T23:44:41.826+0000	finished restoring uwazi_development.sessions (1 document)
2018-10-12T23:44:41.826+0000	restoring indexes for collection uwazi_development.users from metadata
2018-10-12T23:44:41.852+0000	reading metadata for uwazi_development.connections from /home/node/uwazi/blank_state/uwazi_development/connections.metadata.json
2018-10-12T23:44:41.852+0000	reading metadata for uwazi_development.relationtypes from /home/node/uwazi/blank_state/uwazi_development/relationtypes.metadata.json
2018-10-12T23:44:41.852+0000	reading metadata for uwazi_development.templates from /home/node/uwazi/blank_state/uwazi_development/templates.metadata.json
2018-10-12T23:44:42.330+0000	restoring uwazi_development.templates from /home/node/uwazi/blank_state/uwazi_development/templates.bson
2018-10-12T23:44:42.330+0000	no indexes to restore
2018-10-12T23:44:42.330+0000	finished restoring uwazi_development.templates (0 documents)
2018-10-12T23:44:42.331+0000	reading metadata for uwazi_development.dictionaries from /home/node/uwazi/blank_state/uwazi_development/dictionaries.metadata.json
2018-10-12T23:44:42.604+0000	restoring uwazi_development.connections from /home/node/uwazi/blank_state/uwazi_development/connections.bson
2018-10-12T23:44:42.604+0000	restoring indexes for collection uwazi_development.connections from metadata
2018-10-12T23:44:42.892+0000	restoring uwazi_development.relationtypes from /home/node/uwazi/blank_state/uwazi_development/relationtypes.bson
2018-10-12T23:44:42.893+0000	no indexes to restore
2018-10-12T23:44:42.893+0000	finished restoring uwazi_development.relationtypes (0 documents)
2018-10-12T23:44:42.893+0000	reading metadata for uwazi_development.entities from /home/node/uwazi/blank_state/uwazi_development/entities.metadata.json
2018-10-12T23:44:43.213+0000	restoring uwazi_development.dictionaries from /home/node/uwazi/blank_state/uwazi_development/dictionaries.bson
2018-10-12T23:44:43.337+0000	no indexes to restore
2018-10-12T23:44:43.337+0000	finished restoring uwazi_development.dictionaries (0 documents)
2018-10-12T23:44:44.459+0000	finished restoring uwazi_development.users (2 documents)
2018-10-12T23:44:44.703+0000	restoring uwazi_development.entities from /home/node/uwazi/blank_state/uwazi_development/entities.bson
2018-10-12T23:44:44.703+0000	finished restoring uwazi_development.connections (0 documents)
2018-10-12T23:44:44.782+0000	restoring indexes for collection uwazi_development.entities from metadata
2018-10-12T23:44:45.439+0000	finished restoring uwazi_development.entities (0 documents)
2018-10-12T23:44:45.439+0000	done
uwazi-docker: Applyng yarn reindex. This will use data from MongoDB to feed Elastic Search
yarn run v1.5.1
$ node run.js ./database/reindex_elastic.js
Deleting index... uwazi_development
{ FetchError: request to http://elasticsearch:9200/uwazi_development failed, reason: connect ECONNREFUSED <IP_ADDRESS>:9200
    at ClientRequest.<anonymous> (/home/node/uwazi/node_modules/node-fetch/index.js:133:11)
    at emitOne (events.js:96:13)
    at ClientRequest.emit (events.js:188:7)
    at Socket.socketErrorListener (_http_client.js:310:9)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at emitErrorNT (net.js:1296:8)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)
  name: 'FetchError',
  message: 'request to http://elasticsearch:9200/uwazi_development failed, reason: connect ECONNREFUSED <IP_ADDRESS>:9200',
  type: 'system',
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED' }
Creating index... uwazi_development
{ FetchError: request to http://elasticsearch:9200/uwazi_development failed, reason: connect ECONNREFUSED <IP_ADDRESS>:9200
    at ClientRequest.<anonymous> (/home/node/uwazi/node_modules/node-fetch/index.js:133:11)
    at emitOne (events.js:96:13)
    at ClientRequest.emit (events.js:188:7)
    at Socket.socketErrorListener (_http_client.js:310:9)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at emitErrorNT (net.js:1296:8)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)
  name: 'FetchError',
  message: 'request to http://elasticsearch:9200/uwazi_development failed, reason: connect ECONNREFUSED <IP_ADDRESS>:9200',
  type: 'system',
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED' }
Indexing documents and entities... - 0 indexed
Done, took 0.04 seconds
Done in 14.20s.
uwazi-docker: If no fatal errors occurred, you will not need to use this command again

# fititnt at bravo in ~/tmp/uwazi-docker on git:master o [20:45:05]
$ code .

# fititnt at bravo in ~/tmp/uwazi-docker on git:master o [20:47:16]
$ code .

# fititnt at bravo in ~/tmp/uwazi-docker on git:master o [20:47:57]
$ docker-compose run -e IS_FIRST_RUN=true --rm uwazi
Starting uwazi-docker_elasticsearch_1 ... done
Starting uwazi-docker_mongo_1         ... done
uwazi-docker: IS_FIRST_RUN: true
uwazi-docker: Enviroment variable IS_FIRST_RUN is true. Assuming need to install database from blank state

uwazi-docker: Deleting mongo uwazi_development MongoDB database
MongoDB shell version v3.6.4
connecting to: mongodb://mongo:27017/uwazi_development
MongoDB server version: 3.4.14
WARNING: shell and server versions do not match
{ "dropped" : "uwazi_development", "ok" : 1 }

uwazi-docker: Importing /home/node/uwazi/blank_state/uwazi_development to mongo uwazi_development MongoDB database
2018-10-12T23:48:07.376+0000	the --db and --collection args should only be used when restoring from a BSON file. Other uses are deprecated and will not exist in the future; use --nsInclude instead
2018-10-12T23:48:07.376+0000	building a list of collections to restore from /home/node/uwazi/blank_state/uwazi_development dir
2018-10-12T23:48:07.377+0000	reading metadata for uwazi_development.translations from /home/node/uwazi/blank_state/uwazi_development/translations.metadata.json
2018-10-12T23:48:07.551+0000	restoring uwazi_development.translations from /home/node/uwazi/blank_state/uwazi_development/translations.bson
2018-10-12T23:48:07.552+0000	reading metadata for uwazi_development.users from /home/node/uwazi/blank_state/uwazi_development/users.metadata.json
2018-10-12T23:48:07.552+0000	reading metadata for uwazi_development.sessions from /home/node/uwazi/blank_state/uwazi_development/sessions.metadata.json
2018-10-12T23:48:07.552+0000	reading metadata for uwazi_development.settings from /home/node/uwazi/blank_state/uwazi_development/settings.metadata.json
2018-10-12T23:48:07.553+0000	no indexes to restore
2018-10-12T23:48:07.553+0000	finished restoring uwazi_development.translations (1 document)
2018-10-12T23:48:07.553+0000	reading metadata for uwazi_development.connections from /home/node/uwazi/blank_state/uwazi_development/connections.metadata.json
2018-10-12T23:48:07.749+0000	restoring uwazi_development.users from /home/node/uwazi/blank_state/uwazi_development/users.bson
2018-10-12T23:48:08.111+0000	restoring uwazi_development.connections from /home/node/uwazi/blank_state/uwazi_development/connections.bson
2018-10-12T23:48:08.161+0000	restoring indexes for collection uwazi_development.connections from metadata
2018-10-12T23:48:08.358+0000	restoring uwazi_development.settings from /home/node/uwazi/blank_state/uwazi_development/settings.bson
2018-10-12T23:48:08.556+0000	restoring uwazi_development.sessions from /home/node/uwazi/blank_state/uwazi_development/sessions.bson
2018-10-12T23:48:08.583+0000	restoring indexes for collection uwazi_development.users from metadata
2018-10-12T23:48:09.058+0000	no indexes to restore
2018-10-12T23:48:09.058+0000	finished restoring uwazi_development.settings (1 document)
2018-10-12T23:48:09.058+0000	restoring indexes for collection uwazi_development.sessions from metadata
2018-10-12T23:48:09.059+0000	reading metadata for uwazi_development.relationtypes from /home/node/uwazi/blank_state/uwazi_development/relationtypes.metadata.json
2018-10-12T23:48:09.256+0000	finished restoring uwazi_development.sessions (1 document)
2018-10-12T23:48:09.256+0000	reading metadata for uwazi_development.templates from /home/node/uwazi/blank_state/uwazi_development/templates.metadata.json
2018-10-12T23:48:09.421+0000	restoring uwazi_development.relationtypes from /home/node/uwazi/blank_state/uwazi_development/relationtypes.bson
2018-10-12T23:48:09.421+0000	no indexes to restore
2018-10-12T23:48:09.421+0000	finished restoring uwazi_development.relationtypes (0 documents)
2018-10-12T23:48:09.421+0000	finished restoring uwazi_development.connections (0 documents)
2018-10-12T23:48:09.421+0000	reading metadata for uwazi_development.dictionaries from /home/node/uwazi/blank_state/uwazi_development/dictionaries.metadata.json
2018-10-12T23:48:09.421+0000	reading metadata for uwazi_development.entities from /home/node/uwazi/blank_state/uwazi_development/entities.metadata.json
2018-10-12T23:48:09.569+0000	finished restoring uwazi_development.users (2 documents)
2018-10-12T23:48:09.569+0000	restoring uwazi_development.templates from /home/node/uwazi/blank_state/uwazi_development/templates.bson
2018-10-12T23:48:09.570+0000	no indexes to restore
2018-10-12T23:48:09.570+0000	finished restoring uwazi_development.templates (0 documents)
2018-10-12T23:48:09.730+0000	restoring uwazi_development.dictionaries from /home/node/uwazi/blank_state/uwazi_development/dictionaries.bson
2018-10-12T23:48:09.730+0000	no indexes to restore
2018-10-12T23:48:09.730+0000	finished restoring uwazi_development.dictionaries (0 documents)
2018-10-12T23:48:09.904+0000	restoring uwazi_development.entities from /home/node/uwazi/blank_state/uwazi_development/entities.bson
2018-10-12T23:48:09.907+0000	restoring indexes for collection uwazi_development.entities from metadata
2018-10-12T23:48:10.195+0000	finished restoring uwazi_development.entities (0 documents)
2018-10-12T23:48:10.195+0000	done
uwazi-docker: Applyng yarn reindex. This will use data from MongoDB to feed Elastic Search
yarn run v1.5.1
$ node run.js ./database/reindex_elastic.js
Deleting index... uwazi_development
Creating index... uwazi_development
Indexing documents and entities... - 0 indexed
Done, took 1.776 seconds
Done in 6.04s.
uwazi-docker: If no fatal errors occurred, you will not need to use this command again

# fititnt at bravo in ~/tmp/uwazi-docker on git:master o [20:48:17]
$ docker-compose up -d uwazi
uwazi-docker_mongo_1 is up-to-date
uwazi-docker_elasticsearch_1 is up-to-date
Creating uwazi-docker_uwazi_1 ... done

# fititnt at bravo in ~/tmp/uwazi-docker on git:master o [20:49:06]
$ docker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                              NAMES
940816c3853c        uwazi-docker_uwazi                                    "/docker-entrypoint.…"   4 minutes ago       Up 4 minutes        <IP_ADDRESS>:3000->3000/tcp             uwazi-docker_uwazi_1
8cefdf983acb        mongo:3.4                                             "docker-entrypoint.s…"   9 minutes ago       Up 9 minutes        <IP_ADDRESS>:27017->27017/tcp           uwazi-docker_mongo_1
88b856a30df6        docker.elastic.co/elasticsearch/elasticsearch:5.5.3   "elasticsearch -Expa…"   9 minutes ago       Up 9 minutes        <IP_ADDRESS>:9200->9200/tcp, 9300/tcp   uwazi-docker_elasticsearch_1

Ahh, hey @vasyugan, maybe this specific issue is more simple, even if I cannot reproduce.
Look that at the creation of the script, instead of docker volumes, the uwazi-docker store the data of the mongo at a folder ./data the cloned repository, so is a very good possibility that it is just a filesystem permission issue.
Reference image:

If you could debug a bit more maybe I could document it if is not something very, very specific to where you are running uwazi-docker.

@fitint Thanks, but I am running the docker-compose command through sudo, so it has root permissions. Therefore it is difficult to see how this could hinge on permissions. Also, couldn't reproduce this issue, got other instead...

Ah ok.
But just to mention, at least on my system (Docker version 18.03.1-ce, build 9ee9f40 / Ubuntu 16.04.4 LTS) and I guess in other Ubuntu 16 on a remote VPS, I was able to do the full testing with a non-root user that is allowed to manage docker (this depends of your setup), but since the docker itself runs as root, the files created by the docker on ./data folder are with root permission, so I only need root permission to full unninstall that clean up all data (see https://github.com/fititnt/uwazi-docker/blob/master/uninstall.md).
There are several ways to run docker containers, and was choosen to store the important data on this folder and not as docker volume to give a bit more chance for someone do not delete all data by accident and also because can be a bit more simple to make a backup and (if someone start with docker and later want to make full change to a production system) make a bit easier. That is why, by default, the data is not stored as docker volumes, but is possible to do it. I could pass links or document more on that.

well, for now this is how I got around it: rm -rf the data directory, added myself to the docker group and restarted the whole process as non-root. Not sure what got me into this mess in the first place, but I'd tend to close the issue for now.

No problem. Docker is a nice tool to know. What you learn here will serve you for other projects. Also, I will help you a but more testing and with how to make pull requests, so you also lean how to contribute.
The most hard part tend to be the time to discover and test, so if you already know how to make pull requests, will make it easier. We can start with the issue #15.


[BEAM-11815] DynamoDBIO should read all data with limits

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @iemejia PTAL

Run Java PreCommit

amazing. thanks @aromanenko-dev


Selection of multiple flights to edit/delete in slot table

Feature request:
Possibility to select multiple flights to remove/edit in slot table

Won't do this for the moment. I will keep this issue open in case I do plan on adding Filament (#432).
In the meantime, I do accept PR's should you really want to add this in.


examples/sft/run_peft.sh model load dtype error

System Info
peft installed from main
c4c826c0c0dfe06620db8b8adcfa55f22acfed61

transformers version: 4.39.1
Platform: Linux-6.5.0-26-generic-x86_64-with-glibc2.35
Python version: 3.11.8
Huggingface_hub version: 0.22.0
Safetensors version: 0.4.2
Accelerate version: 0.28.0
PyTorch version (GPU?): 2.2.1+cu121 (True)
GPU RTX3090

Who can help?
@pacman100 @younesbelkada
Information

[X] The official example scripts
[ ] My own modified scripts

Tasks

[X] An officially supported task in the examples folder
[ ] My own task or dataset (give details below)

Reproduction
CUDA_VISIBLE_DEVICES=0 python examples/sft/train.py \
--seed 100 \
--model_name_or_path "mistralai/Mistral-7B-v0.1" \
--dataset_name "smangrul/ultrachat-10k-chatml" \
--chat_template_format "chatml" \
--add_special_tokens False \
--append_concat_token False \
--splits "train,test" \
--max_seq_len 2048 \
--num_train_epochs 1 \
--logging_steps 5 \
--log_level "info" \
--logging_strategy "steps" \
--evaluation_strategy "epoch" \
--save_strategy "epoch" \
--bf16 True \
--packing True \
--learning_rate 1e-4 \
--lr_scheduler_type "cosine" \
--weight_decay 1e-4 \
--warmup_ratio 0.0 \
--max_grad_norm 1.0 \
--output_dir "mistral-sft-lora" \
--per_device_train_batch_size 8 \
--per_device_eval_batch_size 8 \
--gradient_accumulation_steps 8 \
--gradient_checkpointing True \
--use_reentrant True \
--dataset_text_field "content" \
--use_peft_lora True \
--lora_r 8 \
--lora_alpha 16 \
--lora_dropout 0.1 \
--lora_target_modules "all-linear" \
--use_4bit_quantization True \
--use_nested_quant True \
--bnb_4bit_compute_dtype "bfloat16" \
--use_flash_attn True

it seems that --bf16 may be having a strange interaction with the bitsandbytes config
adding
--bnb_4bit_quant_storage_dtype "bfloat16"

allows it to run, but that seems like a hack b/c the default for that is typically uint8
Expected behavior
the example script not producing the following error
`low_cpu_mem_usage` was None, now set to True since model is quantized.
Flash Attention 2.0 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in MistralForCausalLM is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `torch_dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", torch_dtype=torch.float16)`
...
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

adding
--bnb_4bit_quant_storage_dtype "bfloat16"

allows it to run, but that seems like a hack b/c the default for that is typically uint8

I don't think this is a hack, bfloat16 is a legitimate value for that parameter. Note that in the script, it's float32 by default, not uint8.

Yea, sorry, "hack" was probably the wrong word. It seems that the default storage type in the bitsandbytes library is uint8 (e.g. https://github.com/TimDettmers/bitsandbytes/blob/main/bitsandbytes/nn/modules.py#L216) and that if someone is quantizing the base model its probable that they would like to use that storage dtype.
However, using `--bnb_4but_quant_storage_dtype "uint8" produces,
ValueError: Can't instantiate MistralForCausalLM model under dtype=torch.uint8 since it is not a floating point dtype

I haven't seen examples of using bfloat16 or float16 or float32 for the bnb storage type but that doesn't mean much! I was wondering if that implies more RAM usage or if the 4bit parameters are efficiently packed into whatever storage dtype is specified.

I think maybe,
quant_storage_dtype should default to uint8 and be passed to bnb_config here (as is already the case)
https://github.com/huggingface/peft/blob/main/examples/sft/utils.py#L107
but it shouldn't be reused when creating the model here
https://github.com/huggingface/peft/blob/main/examples/sft/utils.py#L133
you can probably set torch_dtype to None if a bnb_config exists and/or add it to the script args and parse it and/or get it from --bf16 or --fp16` if they are passed (which currently seem to be ignored)

Thanks a lot for investigating further. Your suggestion sounds reasonable to me, but I'm not an expert on this matter. Pinging @pacman100 who wrote this script if he thinks that this is a good solution.

Hello @galtay, yes, you are correct, I have fixed this in https://github.com/pacman100/DHS-LLM-Workshop/blob/ca763838b6b87fe9b1d08961ce8b8295f7ecd2b0/chat_assistant/sft/training/utils.py#L148-L155 but forgot to make the same changes in the PEFT example.

I have raised the above PR with the fixes and added @galtay as the co-author. Could you please confirm if this resolves the issue?

if the 4bit parameters are efficiently packed into whatever storage dtype is specified

This is the case but I too have observed more memory usage with FP32 dtype.

thanks @pacman100 ! i'll do some tests tomorrow


Merge 4.10.x to  5.3.x

CLA signed by all contributors.

So this PR accidentally got merged to master, and skipped 5.3.x branch.


Update deprecated method - Scenario.embed() to Scenario.attach() "Fixes #486"

5.7.0 (2020-05-01)
Added
[Java] Scenario.log(String) & Scenario.attach(byte[], String, String) (#1893 Tim te Beek)
Deprecated
[Java] Scenario.write(String) & Scenario.embed(byte[], String, String) (#1893 Tim te Beek)

@luke-hill, thanks for looking into this. I looked through the docs and could find only one other instance of embed, for Ruby. I have updated the code, but I am not familiar with Ruby. So please review the change.
4.0.0.rc.4
Added
Methods log and attach can be used in step definitions to attach text or images
Deprecated
embed in step definitions in favor of attach (cucumber#897)

Does this fix #486? If so, please make sure that gets closed on merge (iirc you can do so automatically by adding "Fixes #486" to the name of the PR)

Hi @josepj,
Thanks for your making your first contribution to Cucumber, and welcome to the Cucumber committers team! You can now push directly to this repo and all other repos under the cucumber organization! 🍾
In return for this generous offer we hope you will:

✅ Continue to use branches and pull requests. When someone on the core team approves a pull request (yours or someone else's), you're welcome to merge it yourself.
💚 Commit to setting a good example by following and upholding our code of conduct in your interactions with other collaborators and users.
💬 Join the community Slack channel to meet the rest of the team and make yourself at home.
ℹ Don't feel obliged to help, just do what you can if you have the time and the energy.
🙋 Ask if you need anything. We're looking for feedback about how to make the project more welcoming, so please tell us!

On behalf of the Cucumber core team,
Aslak Hellesøy
Creator of Cucumber

Hi @josepj,
Thanks for your making your first contribution to Cucumber, and welcome to the Cucumber committers team! You can now push directly to this repo and all other repos under the cucumber organization! 🍾
In return for this generous offer we hope you will:

✅ Continue to use branches and pull requests. When someone on the core team approves a pull request (yours or someone else's), you're welcome to merge it yourself.
💚 Commit to setting a good example by following and upholding our code of conduct in your interactions with other collaborators and users.
💬 Join the community Slack channel to meet the rest of the team and make yourself at home.
ℹ Don't feel obliged to help, just do what you can if you have the time and the energy.
🙋 Ask if you need anything. We're looking for feedback about how to make the project more welcoming, so please tell us!

On behalf of the Cucumber core team,
Aslak Hellesøy
Creator of Cucumber


portals.conf(5) not installed

https://search.nixos.org/options?channel=unstable&show=xdg.portal.config says I'm supposed to check portals.conf(5) but apropos portals.conf returns nothing. Is there something wrong with the manpage?
nixos-unstable
Originally posted by @sagehane in https://github.com/NixOS/nixpkgs/issues/247766#issuecomment-1825120120

Man pages are compressed by default, could this be having an effect. You can check it exists by finding the store entry and looking for the man pages.

Just checked and /nix/store/hgynmw3z684rcfk1418kdll49py8qkij-xdg-desktop-portal-1.18.1/share/man/man5/portals.conf.5.gz is a valid path on my setup but it doesn't seem to symlink into /run/current-system/sw/share/man/man5.
MANPATH=/nix/store/hgynmw3z684rcfk1418kdll49py8qkij-xdg-desktop-portal-1.18.1/share/man man portals.conf works fine.

@sagehane When you did nixos-rebuild did you restart your machine? Sometimes that can have an effect.

@sagehane When you did nixos-rebuild did you restart your machine? Sometimes that can have an effect.

I've rebooted like 2-3 times while testing the new XDG config stuff. I've also rebuilt the system about 5 times too. Everything other than the manpage seems to work at a glance. Maybe the store is corrupted and that caused issues? I doubt it.

I suppose if we can reproduce in a vm then it definitely is a bug with the gnome setup that needs to be looked into. I haven't been able to build a vm since it crashes on boot probably for some unrelated reason. I don't have any idea what needs to be fixed here unfortunately.

Yes, the issue is that the file is not installed (linked), we currently only link subset of files. #109060 might fix that.

@jtojnar can you comment on whether #268197 fixes this bug for you?

Yeah, cannot reproduce any more – that was probably the fix.


[Bug] Separator not imported

First, I should say that I can run spleeter just fine from the command line and it works as expected (at least on the examples I've tried so far), but when I try to run it from an IDE (PyCharm), it doesn't work.  The code I'm using is
from spleeter.separator import Separator

print('Separator imported')#the code never reaches this line

separator = Separator("spleeter:2stems")
separator.separate_to_file("test.wav", "/output")

The program hangs on the first line and no error message is ever displayed. I'm not a Python expert so I've no clues as to what might've caused this bahaviour.
The wiki page dealing with this can be found here https://github.com/deezer/spleeter/wiki/4.-API-Reference
Running import spleeter.separator.Separator in a Python console, produces the following error
AttributeError: partially initialized module 'pandas' has no attribute 'core' (most likely due to a circular import)
I'm running this on Windows 10 (64 bit), PyCharm 2019.3 (community edition)

Solved!
This appears to be a permission issue and I solved this by running PyCharm with administrator privilages. I suspect that some of the import dependencies referenced by the Separator package can't be accessed with lower privilages.
Anyway, I hope this could be useful to people experiencing this issue.

Solved!
This appears to be a permission issue and I solved this by running PyCharm with administrator privilages. I suspect that some of the import dependencies referenced by the Separator package can't be accessed with lower privilages.
Anyway, I hope this could be useful to people experiencing this issue.

Hello, is the spleeter running normally through pycharm in win10? The command-line mode is OK, but I want to run it through the run mode in pycham, which is also convenient to run spleeter in batch mode


remove unused syn features

We'll see what CI says but cargo check --tests passes if I remove these features, and based on what the syn docs say the code should just fail to compile if these features are needed

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 81.5%. Comparing base (af6930d) to head (967cad2).

Additional details and impacted files
@@            Coverage Diff            @@
##           master    #1325     +/-   ##
=========================================
- Coverage    81.6%    81.5%   -0.1%
=========================================
  Files         867      867
  Lines      368900   368900
=========================================
- Hits       301052   301016     -36
- Misses      67848    67884     +36


Need HTTPS for Behance and Dribble links to work

The Behance and Dribble links were broken. Without the https the link will just lead to a 404.

Thank you for the update! ☺


docker transparency workspace bind mounts

If you need to mount folders within the dev container into your own containers using docker-outside-of-docker and you do not want to use Docker in Docker  to avoid a potential performance penalty.
This is possible by making use of the hosts local workspace value as bind mount location inside the container.
This is possible without making use of Docker in Docker to avoid performance penalties.
Like In GitHub Codespaces, the workspace folder is **available in the same place on the host as it is in the container,**so you can bind workspace contents as you would normally by doing as following:
https://github.com/DavidBerichon/templates/pull/1

Left some thoughts on you PR --> https://github.com/DavidBerichon/templates/pull/1


Bug 2180666: Display PVC size correctly in Add disk modal

📝 Description
Fixes:
https://bugzilla.redhat.com/show_bug.cgi?id=2180666
Display Persistent Volume Claim size correctly, readable for a user, in Add disk modal accessible from VM Disks page, when choosing PVC (creates PVC) for Source field. Display the value in appropriate size and unit (for example GiB instead of big number + B as bytes unit).
🎥 Demo
The actual info about the selected PVC as an example of the expected size when selecting that PVC later in Add disk modal:

Before:
Not very user friendly PVC size (in bytes):

After:
Nice, readable size - smaller number with GiB units instead of bytes:

@avivtur @metalice @pcbailey @upalatucci @vojtechszocs please review

/lgtm

/cherry-pick release-4.13


Merry Christmas and happy new year!

Merry Christmas and happy new year to all of you 😄

Happy New Year Howard.

Merry Christmas @omeid

Merry Christmas and happy new year @davesters @gertcuykens  @Dr-Terrible @Komosa @smartrevolution  @NoahShen @tjackiw @nickpresta  @suhdev @lanzafame @zhibinr @dchapes @algebraic-brain @bnyeggen @2pi and all

Lol/

;kick #tiedot HouzuoGuo Masshighlight
I would do that if you'll do that on IRC 🐱

it's been a good star so far!


fixed readme; this is a pull request test of Julia's REPL interface

README.md had img instead of image in part of the example code.

Looks great!

Looks great!


Storage loader fix

When init was modified to call d.load() back in https://github.com/lxc/lxd/commit/de19b81a3a53379e5d2bcb4a255c1bbd2e8506cc the intention was that if a driver needed to define it's own load() function then it would override the common one.
However this isn't working as designed, and instead, because inside driverCommon.load() d is always of type driverCommon it is always calling the common load function which does nothing.
This change pushes the call to the driver specific load function to the main driver Load() function where the type of the driver is the interface rather than the common driver.
This will allow the cephfs load() and future driver's to work correctly.

@stgraber now that the cephfs load function is being called, it seems to be failing the tests about a missing rbd tool on jenkins. Even when running non-cephfs backend tests.

It's a typo, the tool is called rbd not rdb.


remove annoying console.log statements in the code

when you open chrome dev tool (or firefox dev tool or edge) you can see a lot of console stuff
which is better not to be there

having trouble pushing the code
remote: Permission to hasadna/open-bus-map-search.git denied to omkarFintaar.

Hey, @shootermv can you please review my PR and approve it.

hi @Abhid1239 !
glad to see you in our project!
For create a PR, you need:

create a fork (by pressing fork button at github repo)
clone a project on your machine (git clone https://github.com/hasadna/open-bus-map-search)
connect you local project to remote repo (git remote set-url origin https://github.com/YOUR-NICKNAME/open-bus-map-search)
create a branch with your changes
only then you will be able to create a PR
please read contribution section (or some [how to contribute to open source](https://medium.com/@moshevilner/useful-git-tips-when-making-your-first-contribution-or-importance-to-know-rebase-command-ac696c807b24) guide)

Hey @shootermv thanks for the detailed info.
Have created the the branch first and then created the PR please check

thanks @Abhid1239 ! :clap:


Active class on header links

Add an active class on a list-item of the header-nav in documentation website so that we always know which page is active (much like hover style)

#1672


Supporting where arguments in arithmetic functions

Description
Many of the basic arithmetic functions in numpy can take a where argument which is not supported by cupy.
To Reproduce
import cupy as np
a = np.random.randn(10)
b = np.random.randn(10)
np.divide(a, b, where = a<0)

Installation
Conda-Forge (conda install ...)
Environment
# Paste the output here

Additional Information
No response

Hi @the-moliver, just sent a PR for that! #7281


Chaining calculators

Consider the following two real scenarios.
This refines some earlier discussion about chaining calculators here: https://github.com/ixdat/ixdat/issues/147
Scenario 1:
ecms = Measurement.read(...)
if condition1:
    ecms.calibrate(RE_vs_RHE=0.715)
if condition2:
    ecms.calibrate(R_Ohm=125)
t, U = ecms.grab("potential", tspan=[100, 200])

Scenario 2:
ecms = Measurement.read(...)
if condition1:
    ecms.add_calculator(MSCalibration("H2", "M2", F=0.21))
if condition2:
    ecms.set_background("M2", tspan=[20, 30])
t, n_dot_H2 = ecms.grab("n_dot_H2", tspan=[100, 200])

In both scenarios, we have two calculators that may or may not be implemented, before we grab a quantity that can be affected by both of the two calculators. The desired behavior is:

In the case that neither calculator is implemented, i.e. condtion1 is False and condition2 is False, in the first scenario we still want the method to return something: U as the raw potential vs the reference electrode. In the second scenario, we don't want it to return anything but instead raise a QuantificationError, because no sensitivity factor is available with which it could calculate the hydrogen flux.
In the case that condition1 is True and condition2 is False, both scenarios should return something: U is potential vs RHE but not corrected for ohmic drop, and n_dot_H2 is the apparent hydrogen flux without background subtraction.
In the case that condition1 is True and condition2 is False, U is potential vs RHE and corrected for Ohmic drop and n_dot_H2 is background-subtracted hydrogen flux.

This desired behavior is intuitive enough, but the following issues in implementation arise

In ixdat 0.2.x, Measurement.grab looks up string (potential or n_dot_H2) in itself, i.e. it passes it on to Measurement.__getitem__. Assuming it's not in the cache, __getitem__ passes the string on to Measurement.get_series which looks first at series_constructors (are these actually different from Calculators?), and then in its list calibrations (which is by default the same as its calibration_list and will in any case be renamed "calibration"-->"calculator"), looping through the calibrate_series methods, passing the same string on until one is able to return something. When a calibration is added to a Measurement, it goes onto the start of its calibration_list, such that the more recent calibration always gets the first chance of returning a calibrated series.

One problem is that this doesn't work when chaining calibrations which provide calibrated series of the same name. In the first scenario, with both calculators applied, without a further complication in the implementation, you'd get a U which is corrected for ohmic drop but not calibrated to the RHE scale, since the string potential is used regardless of the values of condition1 and condition2. In ixdat 0.2.x the further complication in the implementation is to ensure that chainable calibrations get combined before they are used. The combination of chainable calibrations is the difference between calibration_list and the property calibrations. Combining chainable calibrations at use time is so far only implemented for ECCalibration (solving Scenario 1): ECCalibration.get_series knows to check whether it has RE_vs_RHE or R_Ohm or neither or both, and knows what to do in each case.
Finally, there's the problem is that it's hard to specify if you just don't want a certain calculation done in a chain. This would be the case both if you wanted a calibrated H2 flux without background subtraction after applying both calculators in Scenario 2, or if you wanted a potential on the RHE scale but without ohmic drop correction after applying both calculators in Scenario 1. In 0.2.x, here's no solution for Scenario 1. For Scenario 2, in 0.2.x, there's an argument to grab, as well as grab_signal and grab_flux called remove_background which is a boolean determining whether to remove the background, and the calculation is done in the grab function, which is anathema to the calculators paradigm :( .

The need to go through strings(?) Considering where it is decided which calculators to apply:

We want a Measurement.plot() function which, without input, does the most natural thing, which in general will be the calibrated values if calculators available and raw values if not. We want this not least so that plot() is also a way to see if the Measurement has calibrated data. We want grab to do the same thing (both by default and evt with arguments) as plot, so that you can grab exactly the data you see. Its easier for the user to remember arguments (like remove_background=True) than some kind of string encoding (like f"{item}_bg_removed"). plot should therefore always get data via grab. Simple enough.
But a problem arises when grab tries to get data with potentially chained calculators, from the following facts:

grab can take arguments, but __getitem__ only takes a string
We want the Calculator to do the calculation.
Calculators are called by __getitem__.
Therefore, anything we input as an argument to plot and grab needs to be converted to a string for __getitem__ , which the relevant Calculator recognizes.
Calculators, in their available_series_names property should know all the series they can return.

The problem arising from the above steps is that, with chaining Calculators, this could in principle require the available_series_names reading the available_series_names property of other Calculators connected to the same Measurement. This would be a disaster to implement, not least due to the danger of infinite loops :(

Maybe grab should decide what to do rather than __getitem__ just turning power over to the Calculators? @KennethNielsen had an idea of allowing a list of calculator list as inputs to grab but I can't find the notes on that suggestion now.

I think that basically summarizes what I'm struggling with. Comments welcome!  Until then I think I'll just code on the [calculators] branch until I get something that seems good enough.

New solution idea, which should avoid the need for long complex strings in ixdat lookups:

A Measurement object can at most have one Calculator of each Calculator type. They go in a dictionary, not a list.
If the user tries to add a second Calculator of the same type Measurement.add_calculator tries to add them. If addition is not implemented, it raises a TechniqueError.
Addition of Calculators is in general implemented in such a way that a waring is raised if a value is overwritten. For example::
ec = Measurement.read(..., technique="EC")
cal1 = ec.calibrate(RE_vs_RHE=1.0)
cal2 = ec.calibrate(RE_vs_RHE=0.72)  # Raises a warning and overwrites the value
Complex calibration is always done in a Calculator-constructing classmethod, but when simple calibration like the above is done, the object it returns is the calculator formed before addition to others of the same type. These are then available to the user for custom use, as in the next point
Measurement.grab() accepts a calculator list. This temporarily replaces the Measurements calculator list while looking up the desired value. This is done by setting and clearing a Measurement._temp_calculator_list which,  when available, is preferentially returned over _calculator_list by the property Measurement.calculator_list.
Giving Measurement.grab a calculator list, like adding a calculator, clears the cache.
A "-raw" suffix suppresses all use of calculators. This is implemented in __getitem__.

sweet! I like the dictionary implementation, they do help when things get too complicated for a list.
And the option of adding a temporary calculator list is great. It happens occasionally that you'd just want to plot the raw data after calibrating already, and then ixdat is "too smart" for you. Hope this added flexibility will make it more useful generally.


Add @using Microsoft.AspNetCore.Components.Services in _ViewImports.cshtml by default

Is your feature request related to a problem? Please describe.
In a lot of Razor Component apps we'll navigate programatically using IUriHelper. However it's needed to always add the namespace @using Microsoft.AspNetCore.Components.Services at the top of the file or in the _ViewImports.cshtml file. Since it's needed in a lot of applications i'd suggest we add it in the templates by default.
Describe the solution you'd like
Add @using Microsoft.AspNetCore.Components.Services in _ViewImports.cshtml in the template.

I think what's much more likely is to move it to another namespace. I agree that this is worth doing something about and that's good feedback 👍

@rynowak I was expecting it to be in : Microsoft.AspNetCore.Components.Routing.
Maybe put it there since that is already in the _ViewImports file?

This is being resolved by the referenced PR in slightly different way.


[Design] Fast C++ Bit Unpacking

<EMAIL_ADDRESS>Parquet uses Rle and Bitpacking encoding extensively. The usages in Parquet V1 include

Decoding repetition levels to interpret nested structures
Decoding definition levels to interpret nulls
Dictionary ids

It’s crucial to have a fast bit unpacking library for the Parquet reader. In this doc we present the design and lessons learnt from implementing this little lib.
Bulk Read Contiguous Values
SIMD Algorithms
AVX2 and BMI2 are pretty much available on almost all x86/amd64 CPUs nowadays. We are going to use the _pdep_u64 intrinsics in BMI2 and _mm256_cvtepi[x]_epi[y] intrinsics in AVX2. In Lemire’s https://github.com/lemire/LittleIntPacker the bmipacking32.c uses these intrinsics for some bitWidths, but uses non-SIMD implementations for other bit widths. I guess they were faster than using the SIMD intrinsics when the this library was made. But since this was many years ago, what was faster at that time might not be as fast as the intrinsics now. On Intel CPUs, these SIMD intrinsics are faster than 10 years ago while other instructions may not improve as much. In my benchmarks on Intel CoffeeLake, the algorithms using these SIMD intrinsics outperformed all other non-SIMD implementations with manually unrolled loops and plain bit operations like “and”, “or”, and “shiftings”.
However we shall notice that these intrinsics are slower on AMD CPUs. M1 and other ARM processors may also have different costs. Therefore the design for these CPUs would be different. This design is for Intel CPUs that supports AVX2s only.
The actual design is also relevant to the input bit width and output types. The output types we support now are:

uint8_t (mostly for repetition levels and definition levels)
uint16_t (maybe someone uses it?)
uint32_t (mostly for dictionary ids)

The input bit width for a certain output type is in [1, bits in output type]. In Parquet, all bit packed runs contain a multiple of 8 values. In our implementations, we would process a multiple of 8 values at a time. One advantage of this is that we always process the full bytes in every iteration without having to shift bits, which is quite expensive.
Now we talk about the implementation for each output type.
uint8_t
When the output is uint8_t, we process 8 values a time, and the input buffer pointer would move input “bitWidth” bytes. We load the memory into an integer, then use _pdep_u64 to deposit/scatter the bits in this integer to a uint64_t using a predefined mask. For example if input bitWidth is 4 the mask is 0x0f0f0f0f0f0f0f0f. Then the bitpacked value 0x12345678 would be unpacked to 0x0102030405060708.
Note that the bitWidth <= 8. If we cast the input buffer to uint64_t* like this uint64_t val = *reinterpret_cast<const uint64_t*>(inputBuffer) and don’t specially handle the tail, the code might fail with segmentation fault because the last 8 values are less than 64 bits. In the real implementation, we could either directly dereference the input buffer after casting it to uint64*_t or uint32*_t with special handling of last 8 values, or we could use std::memcpy to copy exactly the number of bytes needed. However, I found that it affects the performance a lot sometimes. The clang compiler SOMETIMES fails to optimize the loop using std::memcpy. Let’s look at the disassembly of these two simple functions:
// 266us to decode 8M values
static inline void unpack1(
        const uint8_t *&inputBuffer,
        uint64_t numValues,
        uint8_t *&outputBuffer) {
    uint64_t numBytes = (numValues + 7) / 8;
    auto readEndOffset = inputBuffer + numBytes;

    while (inputBuffer < readEndOffset) {
        uint64_t val = *reinterpret_cast<const uint8_t *>(inputBuffer);
        *(reinterpret_cast<uint64_t *>(outputBuffer)) = _pdep_u64(val, kPdepMask8[1]);
        inputBuffer++;
        outputBuffer += 8;
    }
}

The corresponding assembly code:
0000000100003480 <__ZL15unpack1to8_casthRPKhyyRPh>:
100003480: 55                    pushq	%rbp
100003481: 48 89 e5              movq	%rsp, %rbp
100003484: 48 8b 17              movq	(%rdi), %rdx         #  rdi Function argument #1 (inputBuffer)
100003487: 4c 8d 8a 00 75 6d 00  leaq	7173376(%rdx), %r9   # rdx Function argument #3 (inputBufferLen)
10000348e: 48 8b 0e              movq	(%rsi), %rcx		#  rsi Function argument #2 (inputBuffer)   rcx Function argument #4 (numValue)
100003491: 49 b8 7f 7f 7f 7f 7f 7f 7f 7f movabsq	$9187201950435737471, %r8
10000349b: 0f 1f 44 00 00        nopl	(%rax,%rax)
1000034a0: 48 8b 02              movq	(%rdx), %rax
1000034a3: c4 c2 fb f5 c0        pdepq	%r8, %rax, %rax
1000034a8: 48 89 01              movq	%rax, (%rcx)
1000034ab: 48 83 c2 07           addq	$7, %rdx
1000034af: 48 89 17              movq	%rdx, (%rdi)
1000034b2: 48 8b 0e              movq	(%rsi), %rcx
1000034b5: 48 83 c1 08           addq	$8, %rcx
1000034b9: 48 89 0e              movq	%rcx, (%rsi)
1000034bc: 48 8b 17              movq	(%rdi), %rdx
1000034bf: 4c 39 ca              cmpq	%r9, %rdx
1000034c2: 72 dc                 jb	0x1000034a0 <__ZL15unpack1to8_casthRPKhyyRPh+0x20>
1000034c4: 5d                    popq	%rbp
1000034c5: c3                    retq
1000034c6: 66 2e 0f 1f 84 00 00 00 00 00 nopw	%cs:(%rax,%rax)

Code using memcpy:
// 1998us to decode 8M values
static inline void unpack1_memcpy(
        const uint8_t *&inputBuffer,
        uint64_t numValues,
        uint8_t *&outputBuffer) {
    uint64_t numBytes = (numValues + 7) / 8;
    auto readEndOffset = inputBuffer + numBytes;

    uint64_t val = 0;
    while (inputBuffer < readEndOffset) {
        std::memcpy(&val, inputBuffer, 1);
        *(reinterpret_cast<uint64_t *>(outputBuffer)) =
                _pdep_u64(val, kPdepMask8[1]);
        inputBuffer++;
        outputBuffer += 8;
    }
}

00000001000034d0 <__ZL17unpack1to8_memcpyhRPKhyyRPh>:
1000034d0: 55                    pushq	%rbp
1000034d1: 48 89 e5              movq	%rsp, %rbp
1000034d4: 41 57                 pushq	%r15
1000034d6: 41 56                 pushq	%r14
1000034d8: 41 55                 pushq	%r13
1000034da: 41 54                 pushq	%r12
1000034dc: 53                    pushq	%rbx
1000034dd: 48 83 ec 18           subq	$24, %rsp
1000034e1: 48 8b 1e              movq	(%rsi), %rbx
1000034e4: 48 c7 45 d0 00 00 00 00movq	$0, -48(%rbp)
1000034ec: 40 84 ff              testb	%dil, %dil
1000034ef: 0f 84 61 00 00 00     je	0x100003556 <__ZL17unpack1to8_memcpyhRPKhyyRPh+0x86>
1000034f5: 49 89 d6              movq	%rdx, %r14
1000034f8: 49 89 f7              movq	%rsi, %r15
1000034fb: 41 89 fc              movl	%edi, %r12d
1000034fe: 48 8d 05 0b 0a 00 00  leaq	2571(%rip), %rax  # 100003f10 <__ZL10kPdepMask8>
100003505: 4a 8b 04 e0           movq	(%rax,%r12,8), %rax
100003509: 48 89 45 c0           movq	%rax, -64(%rbp)
10000350d: 49 69 c4 00 a3 0f 00  imulq	$1024768, %r12, %rax
100003514: 48 01 d8              addq	%rbx, %rax
100003517: 48 89 45 c8           movq	%rax, -56(%rbp)
10000351b: 4c 8b 2a              movq	(%rdx), %r13
10000351e: 66 90                 nop
100003520: 48 8d 7d d0           leaq	-48(%rbp), %rdi
100003524: 48 89 de              movq	%rbx, %rsi
100003527: 4c 89 e2              movq	%r12, %rdx
10000352a: e8 e3 07 00 00        callq	0x100003d12 <dyld_stub_binder+0x100003d12>
10000352f: 48 8b 45 d0           movq	-48(%rbp), %rax
100003533: c4 e2 fb f5 45 c0     pdepq	-64(%rbp), %rax, %rax
100003539: 49 89 45 00           movq	%rax, (%r13)
10000353d: 4c 01 e3              addq	%r12, %rbx
100003540: 49 89 1f              movq	%rbx, (%r15)
100003543: 4d 8b 2e              movq	(%r14), %r13
100003546: 49 83 c5 08           addq	$8, %r13
10000354a: 4d 89 2e              movq	%r13, (%r14)
10000354d: 49 8b 1f              movq	(%r15), %rbx
100003550: 48 3b 5d c8           cmpq	-56(%rbp), %rbx
100003554: 72 ca                 jb	0x100003520 <__ZL17unpack1to8_memcpyhRPKhyyRPh+0x50>
100003556: 48 83 c4 18           addq	$24, %rsp
10000355a: 5b                    popq	%rbx
10000355b: 41 5c                 popq	%r12
10000355d: 41 5d                 popq	%r13
10000355f: 41 5e                 popq	%r14
100003561: 41 5f                 popq	%r15
100003563: 5d                    popq	%rbp
100003564: c3                    retq
100003565: 66 2e 0f 1f 84 00 00 00 00 00 nopw	%cs:(%rax,%rax)
10000356f: 90

We can see the implementation using memcpy spends many more instructions calculating the address. In my benchmarks the second implementation took 1998us to decode 8M values, while the first one only took 266us. This might be a clang issue and it does not always happen. But to avoid such possible regression, I chose to use casting in the loop and get the last 8 values using std::memcpy. Btw it seems Lemire’s bmipacking implementation didn’t handle this and always assume there're >=8 bytes left in the buffer.
Note that even though we don't exhaust all the read 64bits in one iteration and the read might not be aligned, it doesn't seem to affect the performance. This may be a result of CPU caching because a cache line is usually (64 bytes) much larger than 8 bytes, and subsequent reads shall read from the cache.
When the input bitWidth is 8, we can just do simple memcpy. However clang does a good job here and using memcpy has the SAME performance as the algorithm using _pdep_u64. To simplify the code I didn’t special case it.
Another takeaway is that manually unrolling the loop does NOT make any difference. There is no data dependency among the loop iterations,  and just using a plain loop can make the code shorter.
uint16_t
bitWidth in [1, 7]: Process 2 * bitWidth bytes (16 values) a time.
Copy 2 * bitWidth bytes into 2 integers, 8 values each. Then call _pdep_u64 on each of the integers to deposit the values from bitWidth to 8 bits wide, and store the 8 * 2 output values to a piece of memory aligned by 16 bytes. Now use _mm256_cvtepu8_epi16 to cast these 16 values in the register and store them back to memory.
Note that using memcpy for  uint16_t doesn’t make much difference that we saw in uint8_t. It is even slightly faster than dereferencing the memory to a uint64_t integer and shifting its bits before the second _pdep_u64.
Another observation is that manually unrolling the loop to process 32 or 64 values in one iteration does NOT make any difference.
bitWidth = 8: Process 2 * bitWidth bytes (16 values) a time.
For this case we just simply call _mm256_cvtepu8_epi16 without _pdep_u64. This is about 10% faster than the above implementation for 1-7 bitWidth.
bitWidth in [9, 15]: Process bitWidth bytes (2 * 4 values) a time.
We read the memory twice in each iteration of the loop. The first time we read ceil(bitWidth / 2) bytes, and the second time the rest that make up the 8 values. E.g. if bitWidth = 9, read 5 bytes to the first integer, and the next 4 bytes to the second integer. Then _pdep_u64 to deposit the 4 values in the first integer to 16 bits directly. Then shift the first integer right for 36 bits, and second integer left 4 bits and or the two result into the 3rd integer, this is to use up the remaining unhandled 4 bits in the first integer. Call _pdep_u64 on the third integer and store the second 4 values to memory.
When the bitWidth is even, we don’t need to shift the integers. However special casing the even bitWidth doesn’t change the performance. To make it simple, we use the above implementation for both odd and even bitWidth.
bitWidth = 16
Just memcpy. It's faster than the above.
Uint32_t
bitWidth in [1, 7]: Process 2 * bitWidth bytes (16 values) a time.
The same as uint16_t.
Manually unrolling does NOT make a difference.
bitWidth = 8 :  Process 8 bytes (8 values) a time.
Directly use _mm256_cvtepu8_epi32 to convert 8 bits to 32 bits.
bitWidth in [9, 15]: Process bitWidth bytes (2 * 4 values) a time.
Same as uint16_t
bitWidth = 16
Directly use _mm256_cvtepu16_epi32 to convert 16 bits to 32 bits.
Other Considerations
Function overloading vs. templates
We could choose to use templated functions with output type as a template parameter, or choose function overloading. There're 2 reasons I chose function overloading other than templates:

Functions for different output types have different implementations. Templated functions are convenient if the implementation logic for different types is identical, but for our library, it's not the case. Using templated functions makes the code harder to read and maintain.
The output types can only be 1, 2, 4, 8 byte integers. We don't need to generate code for the other types.

Class vs. Global Functions In Utility Headers
We could wrap the input and output buffer in a class and provide public functions to unpack the data. However benchmarks show that calling the functions on a class instance is slower than putting them as global functions. This may be related to function dispatch cost and/or inlining. So I chose to put the functions in a header file. The full implementation is about 500 lines of code and is pretty small.
Function Pointers vs. Switches
The performance of these two are about the same. Since for a given column, the input bitWidth and output type are fixed, CPU can predict the correct function or switches and the cost of function dispatching is very minimal.
Benchmark Results
The following benchmark results are from BitUnpackingBenchmark.cpp for unpacking 8M values. The code was compiled by Apple clang version 12.0.5 (clang-1<IP_ADDRESS>), on MacOS with CPU Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz (CoffeeLake). The results are in microseconds(us).

Output Bit Width
Bit Width
arrow
duckdb
fastpforlib
lemire
velox

8
1
2,750
10,430
4,280

292

8
2
2,730
11,660
3,880

307

8
3
2,790
12,590
4,640

317

8
4
2,680
13,890
3,450

334

8
5
3,030
15,470
4,750

361

8
6
3,120
16,780
4,500

460

8
7
3,380
18,110
5,610

487

8
8
2,900
19,150
2,680

521

16
1
2,750
10,660
2,320

549

16
2
2,670
11,770
2,280

585

16
3
2,790
12,770
2,620

709

16
4
2,610
14,130
2,320

751

16
5
3,050
16,840
2,990

820

16
6
3,030
16,850
6,200

890

16
7
3,210
18,080
3,250

1,130

16
8
2,850
19,540
2,380

956

16
9
3,510
21,530
3,520

1,340

16
10
3,570
23,160
3,500

2,100

16
11
3,790
24,290
3,840

2,190

16
12
3,680
25,780
3,600

2,260

16
13
3,970
27,090
3,990

2,580

16
14
3,990
28,940
4,080

2,330

16
15
4,150
30,660
4,330

2,440

16
16
3,230
32,570
2,970

2,370

32
1
2,730
10,740
2,530
1,520
1,620

32
2
2,680
11,670
2,550
1,580
1,650

32
3
2,800
12,820
2,810
1,640
1,700

32
4
2,640
14,110
2,530
1,650
1,730

32
5
3,080
15,510
2,950
1,810
1,830

32
6
3,150
16,680
2,950
2,250
1,900

32
7
3,240
17,810
3,480
2,280
1,920

32
8
2,950
19,140
2,610
2,380
1,960

32
9
3,430
27,230
3,760
2,480
2,060

32
10
3,570
34,630
3,780
2,440
2,090

32
11
3,730
41,070
4,060
2,740
2,220

32
12
3,660
48,670
3,850
2,730
2,240

32
13
3,950
55,820
4,100
3,030
2,330

32
14
3,990
62,250
3,680
3,270
2,580

32
15
4,130
70,350
3,810
3,720
2,470

32
16
3,240
77,600
3,490
3,840
2,340

Read and Skip Values In a RowSet
Sometimes the queries have filters. Velox would read the columns with filters first, and skip the rows that didn’t pass the filters when reading later columns. Our bit unpacking library will support reading non-contiguous values.
(To Be Finished)
Push Down Filters To Bit Packed Data
(To Be Finished)

cc @oerling


Feature request: error bars (basic and continuous)

Hello! I am interested in using error bars in my Visdom plots, ideally the continuous error bars (https://plot.ly/javascript/continuous-error-bars/), but just as well the basic error bars (https://plot.ly/javascript/error-bars/).
I tried adding basic error bars simply by adding,
'error_y': {
    'type': 'data',
    'array': [10.0],
    'visible': True
},

(for example) to scatter() here: https://github.com/facebookresearch/visdom/blob/master/py/init.py#L648.
This works for the first point, but not for appending additional points (i.e. for the first point I get an error bar, but additional points get added to the plot with no error bars).
I was wondering if I might get some direction about what changes need to be made to get this functionality in, since Plotly supports it but it seems that Visdom does not. Do changes need to be made to server.py?
Thanks!

Here's an example: https://github.com/facebookresearch/visdom/issues/18#issuecomment-309280610

Implementing something along the lines of @ajabri's linked method should only require changes to __init__.py, for as long as the error bar information is added both on creation and appending new info, the server should be able to handle the changes


chore: remove unneeded Plugs

These plugs weren't used at all, as we do not serve any static assets nor we
need live reload for anything (as we do not have any form of web UI).

I think that in future we should move phoenix_live_view to development dependencies as well, as we do not use it in production. This would reduce potential attack surface.


refactor(http): replace unsafe read with AsyncRead::read_buf

[X] The commit messages match the guidelines

Disclaimer: Removes unsafe
Differences between:

Hyper unsafe impl

Does not zero out the unused bytes of the buffer

(zeros them earlier when reserving capacity)

Uses .read

The default impl in AsyncRead

Zeros out the unused bytes of the buffer.
Uses .read

The impl AsyncRead for TcpStream (i.e. the one used by Hyper Server)

Does not zero out anything
Does not use .read

Uses .read_bufs

I haven't dug too deeply into the difference between .read and .read_bufs but the benches are within their usual variance (at least on my computer).
Thoughts?

For more context I understand that .read_bufs corresponds to a readv syscall, but I have no idea what the performance characteristics of using that syscall vs .read would be.

It using readv in this case should be nearly identical. The change hasn't been made I guess because read_buf didn't exist when this part of hyper was written. I also didn't want bad implementations of AsyncRead to zero the memory over and over. I guess we could just say "that's a bad transport", but I can still think people could blame hyper...

I have a love hate relationship with generics. At lease as far as I can tell everywhere in Hyper. I.e. Server and HttpConnector we use TcpStream.
There is a possibility that Config currently allows people to add a connector which is not HttpConnector. There is also a possibility someone doesn't use hyper::Server but then they are using the __Proto proxies.
Couple options: 1. "Plug those holes" by making Conn explicitly use a TcpStream. 2. Make the creation of a Connector unsafe. 3. don't use slow transports - add documentation.
I prefer option 1. but this does slightly increase the commitment that hyper will only use tokio transports.
Option 1 does also deprecate what I imagine was a use case for having custom Connectors.

The reason for being generic is that HTTP can be over any transport. That could be TCP, TCP protected by TLS, Unix Domain Sockets, UDP, or anything else. It's a good thing!
The answer is probably to just do what in this PR...

Cool, let me know if there is anything you'd like me to tweak.

I tried out bench this branch against master also, and saw a small (30,000 ns vs 31,000 ns, 5x on each branch) slow down in this branch. I've looked into the impls of AsyncRead and BufMut and things, and I don't have a good guess as why it seems slower.

Yeah I mean.. the benches are weird sometimes. I've found they run faster or slower if I switch tabs in the same terminal. But its all good, if its slower its slower.


RESTful API support and Task Orchestra

This PR has the following changes.

Add a RESTful API for search through items from database(Currently only mongodb supported)
Rewrite mongodb test spec to make the unit test print less test case data.
Drop support for Postgresql, as we discussed, this can reduce the maintainence cost.
Redesign the task scheduling of the scrapers. Scrapers now using the following scheduing mechainsm:

start() method of a scraper add a MAIN task into task queue. MAIN task is scrapping the list of torrents.
task queue is managed by TaskOrchestra class which is injected into each scraper.
task will look current head of queue, and invoke the executeTask(task: Task): Promise<any> method of the scraper, after this promise resolved. it will look schedule a check after a MIN_INTERVAL
if task queue is empty, TaskOrchestra will schedule a CommonTask() (type = MAIN) after MIN_CHECK_INTERVAL or offset of current time and last executed time of MAIN task. it will use the larger one as the scheduling time.
A MAIN task will queue some SUB task.
if MAIN task find a items from current page is new. it will schedule a new MAIN task for the next page until page number reach user setting's MAX_PAGE_NO

fix process event handler bug. Now the application can do clean up job on SIGINT event

Codecov Report

Merging #26 into master will decrease coverage by 10.91%.
The diff coverage is 62.96%.

@@             Coverage Diff             @@
##           master      #26       +/-   ##
===========================================
- Coverage   57.65%   46.73%   -10.92%
===========================================
  Files          13        7        -6
  Lines         196       92      -104
  Branches       19       17        -2
===========================================
- Hits          113       43       -70
+ Misses         68       39       -29
+ Partials       15       10        -5

Impacted Files
Coverage Δ

src/types.ts
100% <ø> (ø)
:arrow_up:

src/main.ts
0% <0%> (ø)
:arrow_up:

src/test/test-samples.ts
100% <100%> (ø)

src/utils/normalize.ts
76.92% <100%> (+1.92%)
:arrow_up:

src/storage/mongodb-store.ts
66.66% <75%> (+46.66%)
:arrow_up:

src/test/fake-config.ts
52.63% <85.71%> (+21.86%)
:arrow_up:

src/storage/database-constant.ts
0% <0%> (-100%)
:arrow_down:

src/entity/Item.ts

src/entity/media-file.ts

src/entity/Team.ts

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e30d158...53159c2. Read the comment docs.


Json data null

My json file output is :
{
"versionCode": 2,
"cancellable": false,
"url": "https://dseaplive.com/app.apk"
}
but app showing json data null. where is the porblem?

Can I see your JSON url?

Closing because of inactivity


SD card not ejected after verification step.

_- Etcher version: 1.0.0-rc.4

Operating system and architecture: iMac17,1running OS X 10.11.6 (15G1421)
Do you see any meaningful error information on DevTools? No

Nice, simple to use product.
It does not actually eject the SD card that it just created, as it states at the end of the verification process.
It's not a big deal, just wanted to bring it to your attention.  In my case it actually helps my workflow since I need to add/edit files on the SD card image.  Maybe that should be a checkbox option?
Thanks for your work on Etcher.

There is a checkbox option for it, in the settings screen ;-)  (cog icon in the upper-right)
Please try updating to 1.0.0-rc.5 and let us know if it's still not working as expected.

Still behaves the same after updating to 1.0.0-rc.5.  Note the 'boot' volume still visible on the desktop above the Etcher window.
I also made sure that 'Auto-unmount on success' was selected.

That's weird. Maybe there is an edge case on Mountutils that we're not catching?
@ilovetogetspamed What kind of drive do you have? Can you send us the output of diskutil info /dev/diskN?

Looks like I'm not able to reproduce this on a MacBook Pro. This is definitely not an issue on mountutils. We need to unmount the drive before being able to flash to it, so we know that its getting unmounted correctly.
@ilovetogetspamed Can you link us to the exact image you're trying to flash? Could it be that the operating system is re-mounting the drive after we unmount it for some reason (e.g. the image contains certain partition types)?

diskutil info /dev/disk2
Device Identifier:        disk2
Device Node:              /dev/disk2
Whole:                    Yes
Part of Whole:            disk2
Device / Media Name:      Built In SDXC Reader
Volume Name:              Not applicable (no file system)
Mounted:                  Not applicable (no file system)
File System:              None
Content (IOContent):      FDisk_partition_scheme
OS Can Be Installed:      No
Media Type:               Generic
Protocol:                 Secure Digital
SMART Status:             Not Supported
Total Size:               8.0 GB<PHONE_NUMBER> Bytes) (exactly 15564800 512-Byte-Units)
Volume Free Space:        Not applicable (no file system)
Device Block Size:        512 Bytes
Read-Only Media:          No
Read-Only Volume:         Not applicable (no file system)
Device Location:          Internal
Removable Media:          Yes
Media Removal:            Software-Activated
Virtual:                  No
OS 9 Drivers:             No
Low Level Format:         Not supported
OS X El Capitan Version 10.11.6  iMac (Retina 5K, 27-inch, Late 2015) 3.2 GHz Intel Core i5

On May 7, 2017, at 2:10 PM, Juan Cruz Viotti<EMAIL_ADDRESS>wrote:
That's weird. Maybe there is an edge case on Mountutils that we're not catching?
@ilovetogetspamed https://github.com/ilovetogetspamed What kind of drive do you have? Can you send us the output of diskutil info /dev/diskN?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/resin-io/etcher/issues/1385#issuecomment-299723773, or mute the thread https://github.com/notifications/unsubscribe-auth/ADHZnKh5NA8whQdIhba-je5kiNSO0p0fks5r3gkrgaJpZM4NRzFx.

Thanks a lot @ilovetogetspamed ! Do you mind sharing the image that you're flashing as well?

I get the same results with both images from https://www.raspberrypi.org/downloads/raspbian/.  Raspian Jessie with Pixel and Raspian Jessie Lite.

On May 8, 2017, at 10:46 AM, Juan Cruz Viotti<EMAIL_ADDRESS>wrote:
Thanks a lot @ilovetogetspamed https://github.com/ilovetogetspamed ! Do you mind sharing the image that you're flashing as well?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/resin-io/etcher/issues/1385#issuecomment-299887771, or mute the thread https://github.com/notifications/unsubscribe-auth/ADHZnO4TYusuoUBDOvPVzXeR-2zwpA2Oks5r3yrQgaJpZM4NRzFx.

Is it possible there's some kind of background service / program on @ilovetogetspamed 's computer that is remounting the freshly-written drive after Etcher has finished flashing it?

If I manually eject the SD card, it does not get automatically remounted.  I have to physically remove the SD card and reinsert it, for it to mount again.
— Before Reinserting:
$ diskutil info /dev/disk2
Could not find disk: /dev/disk2
— After Reinserting:
$ diskutil info /dev/disk2
Device Identifier:        disk2
Device Node:              /dev/disk2
Whole:                    Yes
Part of Whole:            disk2
Device / Media Name:      Built In SDXC Reader
Volume Name:              Not applicable (no file system)
Mounted:                  Not applicable (no file system)
File System:              None
Content (IOContent):      FDisk_partition_scheme
OS Can Be Installed:      No
Media Type:               Generic
Protocol:                 Secure Digital
SMART Status:             Not Supported
Total Size:               8.0 GB<PHONE_NUMBER> Bytes) (exactly 15564800 512-Byte-Units)
Volume Free Space:        Not applicable (no file system)
Device Block Size:        512 Bytes
Read-Only Media:          No
Read-Only Volume:         Not applicable (no file system)
Device Location:          Internal
Removable Media:          Yes
Media Removal:            Software-Activated
Virtual:                  No
OS 9 Drivers:             No
Low Level Format:         Not supported

On May 8, 2017, at 11:30 AM, Andrew Scheller<EMAIL_ADDRESS>wrote:
Is it possible there's some kind of background service / program on @ilovetogetspamed https://github.com/ilovetogetspamed 's computer that is remounting the freshly-written drive after Etcher has finished flashing it?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/resin-io/etcher/issues/1385#issuecomment-299901067, or mute the thread https://github.com/notifications/unsubscribe-auth/ADHZnMINVRCyhzodF_evGfkdYquP5jcBks5r3zUCgaJpZM4NRzFx.

I can reproduce with the Raspbian images. The unmounting does succeed, but the drive gets mounted back like 1 or 2 seconds afterwards :S Running mountutils again on the drive does the trick, but we have to find out why this is the case.

Probably be interesting to see if dd does the same thing? (Can't check myself, as I don't have a Mac)

The ‘official (?)’ documentation:  https://www.raspberrypi.org/documentation/installation/installing-images/mac.md

On May 8, 2017, at 1:25 PM, Andrew Scheller<EMAIL_ADDRESS>wrote:
Probably be interesting to see if dd does the same thing? (Can't check myself, as I don't have a Mac)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/resin-io/etcher/issues/1385#issuecomment-299932890, or mute the thread https://github.com/notifications/unsubscribe-auth/ADHZnB7j0zWN3Z1hfyduNO71ikC_ulu9ks5r31ADgaJpZM4NRzFx.

@lurch dd doesn't do unmounting and such, you have to do that manually, afaik (the raspberry docs seem to reflect that as well)

Sorry, to clarify I specifically meant the "auto-remounting the image after writing has completed" behaviour ;-)

@lurch Oh, well, the OS will attempt to remount it after the handle/descriptor to the unmounted device is closed (can differ between OSs, but Mac OS definitely does that) – and we're basically trying to keep it from doing that

Thanks for your quick work!

On May 12, 2017, at 12:40 AM, Juan Cruz Viotti<EMAIL_ADDRESS>wrote:
Closed #1385 https://github.com/resin-io/etcher/issues/1385 via #1416 https://github.com/resin-io/etcher/pull/1416.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/resin-io/etcher/issues/1385#event-1079376766, or mute the thread https://github.com/notifications/unsubscribe-auth/ADHZnIhXh2TSZktbIuyAC5s-hULu4skyks5r4-KrgaJpZM4NRzFx.


clearTimeout in close event.

Not stoping the connectingTImeout counter on close event will sometime close the wrong websocket.
         ws = new (<any>config.constructor)(wsUrl, protocols);

         connectingTimeout = setTimeout(() => {
             log('timeout');
             ws.close();        // in some situation, ws is a connected websocket.
             emitError('ETIMEDOUT', 'Connection timeout');
         }, config.connectionTimeout);

Some routine like this will happen:

connectionTimeout = 4000, maxRetries = 10, reconnectDelay = 1000
start connect().
assign connectingTimeout as connectingTimeout#1.
websocket connection fails, => handleClose.
1000ms later, reconnect, start connect() again.
assign connectingTimeout as connectingTimeout#2.
connect success, handle open event.
clearTimeout(connectingTimeout#2) <=== note that connectingTimeout#2 is cleared but connectingTimeout#1 is not!
4000ms later, connectingTimeout#1 calls,  ws.close() <==== This will close the connected websocket
reconnection ...

I stop the connectingTimeout in close event, so that it won't close connected ws.

Hi, thanks for contributing.
I've just released a full rewrite of this library and this case should be covered. There are lots of breaking changes, please have a look at the readme.
Regards


Manifest is v2

Recently installed the TaskRecorderScreenshot browser extension. It works, thank you very much.
Encountered two issues with it. First is the browser (Chrome) displaying an error that the extension uses Manifest V2. Second, over the weekend the extension in both browsers disappeared. Not uninstalled, not disabled, disappeared as if never installed in the first place. I've combined these two issues in case they are related.
Environment: Edge 119.0.2151.44 and Chrome 119.0.6045.106 on Windows 10 on an HP laptop.

@simon-swtc, thanks for contacting us. Unfortunately, we're unable to handle Support requests for the product here. To contact Support, consider one of the options described here: Support contact options. In particular, you might want to go to the Dynamics 365 community page to get help with your questions from the Dynamics community.