Predictor values are always the name of the categories column

Sorry if this is a too basic question, I'm trying to learn about text classification and I've been dealing with this issue for a few days now.
I'm following the example you provided:
https://github.com/amaiya/ktrain/blob/master/tutorial-04-text-classification.ipynb
The only difference is that i have my data in CSV format and I use pandas library to load it:
df = pd.read_csv('Historical_EdD_EN-ES.csv', low_memory=False, encoding='utf-8')
df['labels'] = pd.cut(df['RelEditDistance_NOFT'], [0, 0.15, 1], labels=[1, 0], include_lowest=True)
df = df[['Source_NOFT', 'labels']]
So I create the categories (1 and 0) based on the 'RelEditDistance_NOFT' column values. Anything below 0.15 will be category 1 and the rest 0. And 'Source_NOFT' column contains the text sentences.
df.head()

Source_NOFT | labels
View the help system for the Adaptive Keys | 0
Zoom in or zoom out | 0
If the slots are faulty, seek technical assist... | 1
Installing a storage drive into drive bay 3 | 0
Lenovo ThinkServer RD650 server (data node) | 1

Then I follow your code to create the model:
(x_train, y_train), (x_test, y_test), preproc = text.texts_from_df(
train_df=df, text_column='Source_NOFT', label_columns=['labels'], max_features=MAX_FEATURES,
maxlen=MAX_LENGTH, ngram_range=3, preprocess_mode='standard')
model = text.text_classifier('nbsvm', (x_train, y_train), preproc=preproc)
learner:
learner = ktrain.get_learner(model, train_data=(x_train, y_train), val_data=(x_test, y_test))
learner.fit(0.001, 3, cycle_len=1, cycle_mult=2)
and predictor:
predictor = ktrain.get_predictor(learner.model, preproc)
Here's when the problem arises:
data = [ 'This movie was horrible! The plot was boring. Acting was okay, though.',
'The film really sucked. I want my money back.',
'What a beautiful romantic comedy. 10/10 would see again!']
predictor.predict(data)
result:
['labels', 'labels', 'labels']
I was expecting to get the actual category values, 1 or 0.
Could you please give me some hint on what could be happening?
Is there something I'm doing wrong?
Thanks a lot in advance,
Fran.

I believe you need to create 2 columns in your DF: ['postive', 'negative'] and set them to 0 or 1,
df['postive'] = df['labels'].apply(lambda x: 1 if x == 1 else 0)
df['negative'] = df['labels'].apply(lambda x: 1 if x == 0 else 0)

then call:
(x_train, y_train), (x_test, y_test), preproc = text.texts_from_df(
train_df=df, text_column='Source_NOFT', label_columns=['postive', 'negative'], max_features=MAX_FEATURES,
maxlen=MAX_LENGTH, ngram_range=3, preprocess_mode='standard')

then call:
predictor.predict(data)

Thank you, @Bidek56
Yes, the labels are expected to be 1-hot-encoded.  I have corrected the docstring for texts_from_csv and texts_from_df to make this clear.
In addition to @Bidek56 's answer, see also this example notebook for converting integer labels in a Pandas dataframe to be 1-hot-encoded.

Thank you very much for your quick responses @Bidek56 and @amaiya!
That makes sense I guess, I'm going to try right away

Works like a charm now!
Thank you very much again :)


Orchard.Specs: Host app installation misses the latest features

I'm not 100% sure if this intentional (since many of the newer features don't have SpecFlow tests): The OrchardSiteFactory has the basic bindings for SpecFlow tests to set up the host application, GivenIHaveInstalledOrchard is one of them. It creates a clean installation and defines which Core features, themes and modules should be used by the host application.
The list of Core features does not contain Reports and the Module list is missing the following: Orchard.AuditTrail, Orchard.Azure.MediaServices, Orchard.CustomForms, Orchard.Dashboards, Orchard.DynamicForms, Orchard.Glimpse, Orchard.jQuery, Orchard.Media, Orchard.MediaPicker, Orchard.MessageBus, Orchard.OpenId, Orchard.Redis, Orchard.Rules, Orchard.TaskLease.

they are not used so it doesn't matter.


Multiple-line strings don't get escaped when a TOML format included

>>> import toml
>>> import tomlkit
>>> s = """
... string = ['''
... [item]
... a = 1
... ''', '''
... [item]
... a = 2
... ''']
... """
>>> toml.loads(s)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    toml.loads(s)
  File "/.../lib/python3.9/site-packages/toml/decoder.py", line 514, in loads
    raise TomlDecodeError(str(err), original, pos)
toml.decoder.TomlDecodeError: Duplicate keys! (line 7 column 1 char 44)
>>> tomlkit.parse(s)
{'string': ['[item]\na = 1\n', '[item]\na = 2\n']}

another bug:
In [1]: import toml

In [2]: s = """
   ...: about = '''
   ...: hi
   ...:
   ...:     '''
   ...: """
   ...:

In [3]: print(toml.loads(s)['about'])
hi

   - Hello world's hello

This library should have a warning that it shouldn't be used.


Mixin config name

https://imgur.com/bba70626-ab95-45a3-8b9b-74f228a7395d

I see 404, but I guess that you want to say me.
I changed mixin config name and the build will be on curseforge soon.


new(userspace): support split config files

What type of PR is this?
/kind feature
Any specific area of the project related to this PR?
/area engine
What this PR does / why we need it:
This PR introduces support to split Falco config file into multiple files.
Given the length of the current Falco configuration, i think it would be great to be able to split the main Falco config file into multiple pieces (ie: perhaps one for each "logical section").
The implementation is as follows:

only supports 1-level of depth (ie: included config files cannot have configs_files directives), to avoid recursion hells
an exception is thrown if included file does not exist, just like we do for rules_file
same as for rules_file directive, user can specify a folder in the configs_files list. The folder will be then processed in lexicographic order, same way as rules_file.
Included config files CAN override existent key-value pairs
configs_files is not supported when passed from command line -o; this is because it is a sort of preprocessor directive therefore cannot be dynamically injected
Since configs_files directive is parsed right after parsing the main yaml node (ie: the Falco main config), its position does not matter; eg: if you have
a foo.yaml with

foo: foo

then, foo node is overridden even in this case:
configs_files: foo.yaml
foo: bar

to fix #2880, we now install a /etc/falco/config.d/ folder and that folder was added to default config file configs_files directive.

Which issue(s) this PR fixes:

Fixes #2880
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

new(userspace): support split config files

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/milestone 0.38.0

This would require the aforementioned change to the current behavior, plus the need to avoid triggering an exception when an included file is missing (since the default Falco config would always have the includes on the local config, even when not present).

This is now done in latest commit, that also adds some more tests.

WDYT @falcosecurity/falco-maintainers ?
I'd love to gather some feedback around this, this might be a good time since we are just past the 0.37.0 release.

Thanks for the bump, I'll take a look!

What if I pass includes over the cmd options -o, should be all fine right?
Plus could we not allow self referencing the original falco.yaml in the includes. Maybe all of this is already addressed, just double-checking.

Could we include the new "includes" key in the falco.yaml with a description

Yes! I'd love to do that in a follow up PR, where i also split up main config file into many smaller sections eg:

engine.yml
outputs.yml
...

WDYT? Didn't want to do that in this PR to avoid increasing its size and reduce readability.

In addition, would it be possible to allow an empty includes:

It should be already possible, but a test on this would be great indeed! I will add it!

What if I pass includes over the cmd options -o, should be all fine right?

Neat! Didn't think of that, will test and report back!

Plus could we not allow self referencing the original falco.yaml in the includes.

Neat too! I will add a test on this!

In addition, would it be possible to allow an empty includes:

Added a test; it was already working fine.

Plus could we not allow self referencing the original falco.yaml in the includes.

Needed a fix; added a test too!
Thanks for the valuable input!

EDIT: i was wrong, we actually support overriding values like: -o "load_plugins[0]=k8saudit" -o "load_plugins[1]=json" but the includes are loaded too early (ie: as soon as the yaml config is loaded) and therefore -o is not taken into account.

So, what happens when the user tries -o includes.... ? Does it get an error or silently fail?

Does it get an error or silently fail?

Silently fail. I think it must be documented in a follow-up PR (ie: when i split main Falco config into different section-specific configs), above the includes: node in the main Falco config.
After all it makes sense if you see it as a "preprocessor directive".

Does it get an error or silently fail?

Silently fail. I think it must be documented in a follow-up PR (ie: when i split main Falco config into different section-specific configs), above the includes: node in the main Falco config. After all it makes sense if you see it as a "preprocessor directive".

I agree it makes sense. However, it should not be silent, in my opinion.

We can add a warning perhaps?

We can add a warning perhaps?

Yes, that's exactly what I meant.

Implemented.

Agree with entire discussion!

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP, incertum
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP,incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/hold

Reached an agreement on the final design for the feature:

[ ] split test in multiple tests
[ ] support including directories; configs contained will be parsed in lexicographically order

[ ]  split test in multiple tests

:+1:

[ ]  support including directories; configs contained will be parsed in lexicographically order

:+1: :+1: :+1: (pls make sure loading ordering work as same as rules_file, for UX consistency :pray: )

[ ]  rename includes to configs_file to be consistent with https://github.com/falcosecurity/falco/blob/master/falco.yaml#L154 (? WDYT @incertum @Andreagit97 ? my 2c, this is different; rules_file is specifying a list of rule files to be loaded, while includes is specifying a list of config files to be included)

I like it, but not totally :innocent:
rules_file has always been wrong (for clear reasons), but it has been preserved to avoid breaking change. I believe this is a good chance to fix it, so I have a counter-proposal:

(in another PR) add rules_files config option, and make rules_file (the old one) an alias to rules_files. This will improve name consistency and avoid any breaking changes here (I've strong opinions on avoiding breaking changes on this - we may just think about removing the alias when reaching Falco 1.0, but not before for sure).
(this PR) just use configs_files

Also, an optional point: I think it would be a good idea to add /etc/falco/config.d by default as well.

Also, an optional point: I think it would be a good idea to add /etc/falco/config.d by default as well.

Agree! Adding it to the TODO.

(this PR) just use configs_files

I have no strong opinions, if @Andreagit97 and @incertum all agree, we can go with this.

Also, an optional point: I think it would be a good idea to add /etc/falco/config.d by default as well.

Agree! Adding it to the TODO.

(this PR) just use configs_files

I have no strong opinions, if @Andreagit97 and @incertum all agree, we can go with this.

configs_files is good, also like the idea of changing rules_file to rules_files and keeping an alias for backward compatibility.

Thank you! I still think that a command line option/debug log with the merged config would be really useful for us and for our users to debug possible issues

@Andreagit97 done in latest commit! I discovered that we already had a --support cmdline option that already printed the config file. I extended it to:

print the expanded config file
print the parsed yaml structure instead of the whole (as in, read from file) file

Example output:
./userspace/falco/falco --support -c ../falco.yaml  -r ../rules/falco_rules.yaml  | jq -r '.config'
Tue Apr  9 11:10:08 2024: Falco version: 0.38.0-176+aa02401 (x86_64)
Tue Apr  9 11:10:08 2024: Falco initialized with configuration files:
Tue Apr  9 11:10:08 2024:    ../falco.yaml
Tue Apr  9 11:10:08 2024: Included config entry unexistent: /etc/falco/config.d
Tue Apr  9 11:10:08 2024: System info: Linux version 6.5.0-26-generic (buildd@lcy02-amd64-070) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.2.0-4ubuntu3) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.41) #26-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar  5 21:19:28 UTC 2024
Tue Apr  9 11:10:08 2024: Loading rules from file ../rules/falco_rules.yaml
{"configs_files": ["/etc/falco/config.d"], "rules_file": ["/etc/falco/falco_rules.yaml", "/etc/falco/falco_rules.local.yaml", "/etc/falco/rules.d"], "engine": {"kind": "kmod", "kmod": {"buf_size_preset": "4", "drop_failed_exit": "false"}, "ebpf": {"probe": "/home/federico/.falco/falco-bpf.o", "buf_size_preset": "4", "drop_failed_exit": "false"}, "modern_ebpf": {"cpus_for_each_buffer": "2", "buf_size_preset": "4", "drop_failed_exit": "false"}, "replay": {"capture_file": ""}, "gvisor": {"config": "", "root": ""}}, "load_plugins": [], "plugins": [{"name": "k8saudit", "library_path": "libk8saudit.so", "init_config": null, "open_params": "http://:9765/k8s-audit"}, {"name": "cloudtrail", "library_path": "libcloudtrail.so"}, {"name": "json", "library_path": "libjson.so"}], "watch_config_files": "true", "time_format_iso_8601": "false", "priority": "debug", "json_output": "false", "json_include_output_property": "true", "json_include_tags_property": "true", "buffered_outputs": "false", "rule_matching": "first", "outputs_queue": {"capacity": "0"}, "stdout_output": {"enabled": "true"}, "syslog_output": {"enabled": "true"}, "file_output": {"enabled": "false", "keep_alive": "false", "filename": "./events.txt"}, "http_output": {"enabled": "false", "url": "http://some.url", "user_agent": "falcosecurity/falco", "insecure": "false", "ca_cert": "", "ca_bundle": "", "ca_path": "/etc/ssl/certs", "mtls": "false", "client_cert": "/etc/ssl/certs/client.crt", "client_key": "/etc/ssl/certs/client.key", "echo": "false", "compress_uploads": "false", "keep_alive": "false"}, "program_output": {"enabled": "false", "keep_alive": "false", "program": "jq '{text: .output}' | curl -d @- -X POST https://hooks.slack.com/services/XXX"}, "grpc_output": {"enabled": "false"}, "grpc": {"enabled": "false", "bind_address": "unix:///run/falco/falco.sock", "threadiness": "0"}, "webserver": {"enabled": "true", "threadiness": "0", "listen_port": "8765", "listen_address": "<IP_ADDRESS>", "k8s_healthz_endpoint": "/healthz", "ssl_enabled": "false", "ssl_certificate": "/etc/falco/falco.pem"}, "log_stderr": "true", "log_syslog": "true", "log_level": "info", "libs_logger": {"enabled": "false", "severity": "debug"}, "output_timeout": "2000", "syscall_event_timeouts": {"max_consecutives": "1000"}, "syscall_event_drops": {"threshold": ".1", "actions": ["log", "alert"], "rate": ".03333", "max_burst": "1", "simulate_drops": "false"}, "metrics": {"enabled": "false", "interval": "1h", "output_rule": "true", "resource_utilization_enabled": "true", "state_counters_enabled": "true", "kernel_event_counters_enabled": "true", "libbpf_stats_enabled": "true", "convert_memory_to_mb": "true", "include_empty_values": "false"}, "base_syscalls": {"custom_set": [], "repair": "false"}, "falco_libs": {"thread_table_size": "262144"}}

Adding another jq to parse the .config json string:
./userspace/falco/falco --support -c ../falco.yaml  -r ../rules/falco_rules.yaml  | jq -r '.config' | jq
Tue Apr  9 11:10:33 2024: Falco version: 0.38.0-176+aa02401 (x86_64)
Tue Apr  9 11:10:33 2024: Falco initialized with configuration files:
Tue Apr  9 11:10:33 2024:    ../falco.yaml
Tue Apr  9 11:10:33 2024: Included config entry unexistent: /etc/falco/config.d
Tue Apr  9 11:10:33 2024: System info: Linux version 6.5.0-26-generic (buildd@lcy02-amd64-070) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.2.0-4ubuntu3) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.41) #26-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar  5 21:19:28 UTC 2024
Tue Apr  9 11:10:33 2024: Loading rules from file ../rules/falco_rules.yaml
{
  "configs_files": [
    "/etc/falco/config.d"
  ],
  "rules_file": [
    "/etc/falco/falco_rules.yaml",
    "/etc/falco/falco_rules.local.yaml",
    "/etc/falco/rules.d"
  ],
  "engine": {
    "kind": "kmod",
    "kmod": {
      "buf_size_preset": "4",
      "drop_failed_exit": "false"
    },
    "ebpf": {
      "probe": "/home/federico/.falco/falco-bpf.o",
      "buf_size_preset": "4",
      "drop_failed_exit": "false"
    },
    "modern_ebpf": {
      "cpus_for_each_buffer": "2",
      "buf_size_preset": "4",
      "drop_failed_exit": "false"
    },
    "replay": {
      "capture_file": ""
    },
    "gvisor": {
      "config": "",
      "root": ""
    }
  },
  "load_plugins": [],
  "plugins": [
    {
      "name": "k8saudit",
      "library_path": "libk8saudit.so",
      "init_config": null,
      "open_params": "http://:9765/k8s-audit"
    },
    {
      "name": "cloudtrail",
      "library_path": "libcloudtrail.so"
    },
    {
      "name": "json",
      "library_path": "libjson.so"
    }
  ],
  "watch_config_files": "true",
  "time_format_iso_8601": "false",
  "priority": "debug",
  "json_output": "false",
  "json_include_output_property": "true",
  "json_include_tags_property": "true",
  "buffered_outputs": "false",
  "rule_matching": "first",
  "outputs_queue": {
    "capacity": "0"
  },
  "stdout_output": {
    "enabled": "true"
  },
  "syslog_output": {
    "enabled": "true"
  },
  "file_output": {
    "enabled": "false",
    "keep_alive": "false",
    "filename": "./events.txt"
  },
  "http_output": {
    "enabled": "false",
    "url": "http://some.url",
    "user_agent": "falcosecurity/falco",
    "insecure": "false",
    "ca_cert": "",
    "ca_bundle": "",
    "ca_path": "/etc/ssl/certs",
    "mtls": "false",
    "client_cert": "/etc/ssl/certs/client.crt",
    "client_key": "/etc/ssl/certs/client.key",
    "echo": "false",
    "compress_uploads": "false",
    "keep_alive": "false"
  },
  "program_output": {
    "enabled": "false",
    "keep_alive": "false",
    "program": "jq '{text: .output}' | curl -d @- -X POST https://hooks.slack.com/services/XXX"
  },
  "grpc_output": {
    "enabled": "false"
  },
  "grpc": {
    "enabled": "false",
    "bind_address": "unix:///run/falco/falco.sock",
    "threadiness": "0"
  },
  "webserver": {
    "enabled": "true",
    "threadiness": "0",
    "listen_port": "8765",
    "listen_address": "<IP_ADDRESS>",
    "k8s_healthz_endpoint": "/healthz",
    "ssl_enabled": "false",
    "ssl_certificate": "/etc/falco/falco.pem"
  },
  "log_stderr": "true",
  "log_syslog": "true",
  "log_level": "info",
  "libs_logger": {
    "enabled": "false",
    "severity": "debug"
  },
  "output_timeout": "2000",
  "syscall_event_timeouts": {
    "max_consecutives": "1000"
  },
  "syscall_event_drops": {
    "threshold": ".1",
    "actions": [
      "log",
      "alert"
    ],
    "rate": ".03333",
    "max_burst": "1",
    "simulate_drops": "false"
  },
  "metrics": {
    "enabled": "false",
    "interval": "1h",
    "output_rule": "true",
    "resource_utilization_enabled": "true",
    "state_counters_enabled": "true",
    "kernel_event_counters_enabled": "true",
    "libbpf_stats_enabled": "true",
    "convert_memory_to_mb": "true",
    "include_empty_values": "false"
  },
  "base_syscalls": {
    "custom_set": [],
    "repair": "false"
  },
  "falco_libs": {
    "thread_table_size": "262144"
  }
}

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP, incertum, leogr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP,incertum,leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

CI Build / check-engine-checksum (pull_request) Failing after 6s

What

CI Build / check-engine-checksum (pull_request) Failing after 6s

What

encoded checksum: 12cb01b5d2379547819a37a4656f1cef2952893ea1225de79614c673f386f645
current checksum: -
current engine checksum differs from the one encoded in userspace/engine/falco_engine_version.h

:thinking:

Rebased on top of new master :)

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP, incertum
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP,incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP, incertum
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP,incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Still broken :/
https://github.com/falcosecurity/falco/actions/runs/8618044652/job/23619985439?pr=3024
Given that it is not a required job, i'd avoid fighting with it rn; let's see if it disappears once in master.

Found the mistery:
echo $(build/userspace/falco/falco -c ./falco.yaml --version | grep 'Engine:' | awk '{print $2}') $(echo $(build/userspace/falco/falco -c ./falco.yaml --version | grep 'Schema version:' | awk '{print $3}') $(build/userspace/falco/falco -c ./falco.yaml --list --markdown | grep '^`' | sort) $(build/userspace/falco/falco -c ./falco.yaml --list-events | sort) | sha256sum)
Cannot read config file (./falco.yaml): Included config entry not existent: /etc/falco/config.d
Error: std::exception
Cannot read config file (./falco.yaml): Included config entry not existent: /etc/falco/config.d
Error: std::exception
Cannot read config file (./falco.yaml): Included config entry not existent: /etc/falco/config.d
Error: std::exception
Cannot read config file (./falco.yaml): Included config entry not existent: /etc/falco/config.d
Error: std::exception
01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b -

:D

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP, incertum
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP,incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP, incertum
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP,incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

I think i'll adapt the code to do the same for configs_files.

Adapted! Also, moved core logic from yaml_helper to falco_configuration class, and updated tests accordingly.
This allowed to enable falco configs files parsing and merging from cmdline option.

This is now finally ready, also added a test to check that we are able to include configs from cmdline.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP, incertum, leogr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP,incertum,leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, FedeDP, incertum, leogr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Andreagit97,FedeDP,incertum,leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


spark tab in rstudio preview server shows many spark tables and there is no search

We installed sparklyr on rstudio server pro pointing to the cluster using yarn-client.
On inspecting the spark tab in rstudio UI we see many spark objects that are not even created by the logged in user.
Moreover there is no search functionality to search for the object.
Please add search and show only the spark objects created by the logged in user.
Please see the attached screen-shot.

@rvij thanks for the feedback. I'm trying to understand a bit more what would be a good way to implement this. A couple questions: Are there multiple hive databases in your organization? Is the user listed under the table metadata? For the second question, please you run the following code for one of the tables to give us some sense of the type of metadata systems like yours are tracking:
# sc <- spark_connect(master = "")
sparklyr:::df_from_sql(sc, "SHOW TBLPROPERTIES atest_10034")

Hi Javier,
Yes that’s correct, our hadoop cluster has 100s of Hive databases. But the specific user can only query few of them.
Rest of them are not readable, so user cannot query those tables/ hive databases, we use MapR distribution of hadoop.
I get following results form the query.

sparklyr:::df_from_sql(sc, "SHOW TBLPROPERTIES atest_10034")

A tibble: 1 x 1
                         result
                          <chr>

1 transient_lastDdlTime\t1476278357
Since there are large number of objects showing up and no search in the spark tab, so I cannot query for my objects I loaded into spark.
Hope this helps.
Rajeev Vij
From: Javier Luraschi <notifications<EMAIL_ADDRESS>Reply-To: rstudio/sparklyr <reply<EMAIL_ADDRESS>Date: Wednesday, October 12, 2016 at 11:05 PM
To: rstudio/sparklyr <sparklyr<EMAIL_ADDRESS>Cc: Rajeev Vij <rvij<EMAIL_ADDRESS>Mention <mention<EMAIL_ADDRESS>Subject: Re: [rstudio/sparklyr] spark tab in rstudio preview server shows many spark tables and there is no search (#266)
sparklyr:::df_from_sql(sc, "SHOW TBLPROPERTIES atest_10034")

@rvij any chance you know the SQL query you would use to filter the tables assigned to each user? I'll mark this as feature request to, at least, consider searching by table name in the UI. However, at this moment in time, it's less clear to me how sparklyr could filter out the tables created by the given user. Thoughts and suggestions welcome.

Thanks yes please include search feature to start with. In order to form a query, I need to know what does sparklyr show in the “spark tab” in Rstudio server?
Does it show all the hive objects/tables in the hive metadata? If that is the case then why do we need to show it in first place. If user would like to process hive tables using spark then he should explicitly create spark table using sparklyr.
From: Javier Luraschi <notifications<EMAIL_ADDRESS>Reply-To: rstudio/sparklyr <reply<EMAIL_ADDRESS>Date: Thursday, October 13, 2016 at 9:45 AM
To: rstudio/sparklyr <sparklyr<EMAIL_ADDRESS>Cc: Rajeev Vij <rvij<EMAIL_ADDRESS>Mention <mention<EMAIL_ADDRESS>Subject: Re: [rstudio/sparklyr] spark tab in rstudio preview server shows many spark tables and there is no search (#266)
@rvijhttps://github.com/rvij any chance you know the SQL query you would use to filter the tables assigned to each user? I'll mark this as feature request to, at least, consider searching by table name in the UI. However, at this moment in time, it's less clear to me how sparklyr could filter out the tables created by the given user. Thoughts and suggestions welcome.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/rstudio/sparklyr/issues/266#issuecomment-253569719, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ACfWykd3I8XzmmtyDMuJW90YrMGlkG-Eks5qzmAUgaJpZM4KVXtE.

@rvij it currently runs this sql query: "SHOW TABLES" through the HiveContext.sql() for Spark < 2.0 and the SessionContext.sql() for Spark 2.0.
If we find no way to query this from Hive, we could track our own list of created tables but this would reset when the rsession restarts. Thoughts welcomed.

Hi @rvij, we are having a similar issue. We're running RStudio on an edge node and connecting to Spark via sparklyr to a MapR Hadoop cluster. However I'm relatively new to this, here is what we are running to connect to spark:
library(sparklyr)
sc <- spark_connect(master = "spark://node04:7077")

Node04 is a node on the Hadoop cluster that has a spark master running. We connect, see several files that were not created by the user, and get an error when clicking to preview one of the files:
Error in if (ncol(df) > 0) df else invisible(df) :
  argument is of length zero
In addition: Warning message:
In value[[3L]](cond) :
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 4, node02): java.io.IOException: Error getting user info for current user, brett01
	at com.mapr.fs.MapRFileSystem.lookupClient(MapRFileSystem.java:665)
	at com.mapr.fs.MapRFileSystem.lookupClient(MapRFileSystem.java:702)
	at com.mapr.fs.MapRFileSystem.getMapRFileStatus(MapRFileSystem.java:1432)
	at com.mapr.fs.MapRFileSystem.getFileStatus(MapRFileSystem.java:1056)
	at org.apache.hadoop.hive.ql.io.RCFile$Reader.<init>(RCFile.java:1343)
	at org.apache.hadoop.hive.ql.io.RCFileRecordReader.<init>(RCFileRecordReader.java:100)
	at org.apache.hadoop.hive.ql.io.RCFileInputFormat.getRecordReader(RCFileInputFormat.java:57)
	at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:237)
	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208)
	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
	at org.apache.spark.r [... truncated]

Also, does the Spark master need to be localized to the client/edge node, or R-Server localized to the node with the spark master? Any insights to how you configured Sparklyr with RStudio and a MapR distro of Spark will be helpful too, thanks

@rvij search functionality is now implemented in rstudio daily builds (until rstudio 1.1 is released), requires using sparklyr from github (until sparklyr 0.6 gets released to CRAN).

@rvij see https://github.com/rstudio/rstudio/issues/1549, will close this one since this is now tracked under the RStudio repo.


Fix for Issue 117, setMetadata to use partialInstance Update

Fixes #<issue_number_goes_here> (it's a good idea to open an issue first for discussion)

[ ] Tests and linter pass
[ ] Code coverage does not decrease (if any source code was changed)
[ ] Appropriate docs were updated (if necessary)

This pull request contains changes for all review comments on PR#146

@kolodny: this PR replaces another one you reviewed previously.  PTAL.

There is already a system test for instance update using setmetadata method. Do you want me to add something specific to update mask?

@muraliQlogic, can you please resolve the conflicts?

@sduskis  I think that was addressed with https://github.com/googleapis/nodejs-bigtable/pull/167#issuecomment-395846526


Handle more error codes when an RP isn't registered

There's more than one error code returned from unregistered RPs.
Fixes https://github.com/Azure/azure-sdk-for-go/issues/20823

We could also add a string slice to RegistrationOptions to allow callers to add additional non-standard error codes to unblock themselves.  Thoughts?
CC @JeffreyRichter

I'm sure I'm missing context but when I look at #20823, it looks like the error code string was just wrong and that this bug should be fixed.
I can see the slight desire to look for error code strings (which should be compared case-sensitively, by the way - I mention this because I saw a call to EqualFold in #20823 but it looks that was may have been removed). But, since error codes are related to HTTP status code, the thing to check on would be BOTH status code AND error code string.
But, do we really have to add this additional infrastructure?

We do already check the HTTP status code and if it's a 409 then we look for this additional information.
Most services return MissingSubscriptionRegistration (I believe all services are supposed to use this error code when the RP is unregistered).  However, the Quota service is returning MissingRegistrationForResourceProvider and I'm sure we can't change that at this point for compat reasons (I have an email out to Roopesh and you about this).
Adding a slice of additional error codes would let customers unblock themselves if/when a service returns a non-standard error code.  That said, it's probably rare enough that the extra machinery isn't worth it (I don't feel strongly about it, just throwing the idea out there).

Who owns the code that is doing this comparison?
If it's us (the SDK), then we can check for both strings.
If it's the customer, then they can check for the right (or both) strings.

OK, so if you want to check for both strings, that's OK with me. Add a comment.

Adding a slice of additional error codes would let customers unblock themselves if/when a service returns a non-standard error code. That said, it's probably rare enough that the extra machinery isn't worth it (I don't feel strongly about it, just throwing the idea out there).

👍 if there's one standard code, it's reasonable to assume the number of variants is small; this may be the only one. We can patch again if another variant turns up and add API if we come to believe there are many unknown variants.

Hi @jhendrixMSFT ,
Thank you for fixing the issue, and I have another similar case: https://github.com/Azure/azure-sdk-for-go/issues/19900, would you please also check it? Thanks!
The error is
 {
   "error": {
     "code": "Subscription Not Registered",
     "message": "Please register to Microsoft.Security in order to view your security status"
   }
 }

@jhendrixMSFT - Sure, it's already in this issue: https://github.com/Azure/azure-sdk-for-go/issues/19900
With track2 sdk, creating of a Microsoft.Security/pricings@2022-03-01 resource is failing to a newly created subscription without a registered Microsoft.Security provider.
│ PUT https://management.azure.com/subscriptions/[subscription-id]/providers/Microsoft.Security/pricings/VirtualMachines?api-version=2022-03-01
│ --------------------------------------------------------------------------------
│ RESPONSE 404: 404 Not Found
│ ERROR CODE: Subscription Not Registered
│ --------------------------------------------------------------------------------
│ {
│   "error": {
│     "code": "Subscription Not Registered",
│     "message": "Please register to Microsoft.Security in order to view your security status"
│   }
│ }
│ --------------------------------------------------------------------------------
│
│
╵

My mistake, I confused this with issue https://github.com/Azure/azure-sdk-for-go/issues/20823


Revert prepublishOnly to prepubl¡sh

Please makes sure these boxes are checked before submitting your PR, thank you!

[x] Make sure you are merging your commits to master branch.
[x] Add some descriptions and refer relative issues for you PR.
[x] Rebase your commits to make your pull request meaningful.
[x] Make sure that your changes pass npm test, npm run lint and npm run build.

Changes in this pull request

Revert prepublishOnly to prepubl¡sh

LGTM


ElasticSearch image fails to build on Mac, citing invalid reference.

Problem description
ElasticSearch Docker container fails to build on Mac running High Sierra. This is because there is no LSB_RELEASE value for the wget URL to use when running on Mac hosts.
Extra information
I can fix this pretty quickly, just logging this issue for the future merge request.
Stack configuration

Docker setup
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:21:31 2018
 OS/Arch:           darwin/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:29:02 2018
  OS/Arch:          linux/amd64
  Experimental:     false

docker-compose version 1.22.0, build f46880f
docker-py version: 3.4.1
CPython version: 3.6.4
OpenSSL version: OpenSSL 1.0.2o  27 Mar 2018

Docker logs
Attaching to

Not sure if I understand, where do we use this variable?


constructRoutes.domToRoutesConfig ignores containerEl-Attribute on single-spa-router

Hi there
constructRoutes.domToRoutesConfig reads out the passed by attributes from the single-spa-router tag on the lines: https://github.com/single-spa/single-spa-layout/blob/8eb1a0a4041f5ef480c60ea1ca88e3245e948cf8/src/constructRoutes.js#L118-L127
but the "containerEl" option, as listed here: https://single-spa.js.org/docs/layout-definition/#single-spa-router, is missing and therefore ignored. After manually hacking this into the minified version it works as expected.
Thanks in advance and would be great to see this fixed!
Versions used:
single-spa: 5.5.3
single-spa-layout: 1.0.0-beta.5
Setup:
<!DOCTYPE html>
<html lang="en">
<head>
    <base href="/" />
    <meta name="importmap-type" content="systemjs-importmap">

    <template id="single-spa-layout" style="display: none;">
        <single-spa-router mode="history" base="/" containerEl="#spa-container">
            <application name="nav"></application>
            <route path="clients">
                <application name="clients"></application>
            </route>
            <route path="settings">
                <application name="settings"></application>
            </route>
        </single-spa-router>
    </template>
</head>

<body>
    <div class="page-wrapper">
        <main id="spa-container" class="page-content" role="main">
            <!-- Should be inserted here -->
        </main>
    </div>

    <script defer<EMAIL_ADDRESS>    <script defer src="https://unpkg.com/zone.js"></script>
    <script defer<EMAIL_ADDRESS>    <script defer src="https://cdnjs.cloudflare.com/ajax/libs/systemjs/4.0.0/system.min.js"></script>
    <script defer src="https://cdnjs.cloudflare.com/ajax/libs/systemjs/4.0.0/extras/amd.min.js"></script>
    <script defer src="https://cdnjs.cloudflare.com/ajax/libs/systemjs/4.0.0/extras/named-exports.js"></script>
    <script defer src="https://cdnjs.cloudflare.com/ajax/libs/systemjs/4.0.0/extras/named-register.min.js"></script>
    <script defer src="~/scripts/single-spa.min.js"></script>
    <script defer src="~/scripts/single-spa-layout.min.js"></script>
    <script type="systemjs-importmap">
        {
        "imports":
        {
        "clients": "https://localhost:44300/clients/main-es2015.js",
        "settings": "https://localhost:44300/settings/main-es2015.js",
        "nav": "https://localhost:44300/nav/main-es2015.js",
        "single-spa": "/scripts/single-spa.min.js",
        "single-spa-layout": "/scripts/single-spa-layout.min.js"
        }
        }
    </script>
    <script>
        // ensure that all defer scripts are loaded
        window.addEventListener('DOMContentLoaded', function () {
            Promise.all([
                System.import('single-spa'),
                System.import('single-spa-layout'),
            ]).then(function (modules) {
                var singleSpa = modules.shift();
                var singleSpaLayout = modules.shift();

                var routes = singleSpaLayout.constructRoutes(document.querySelector('#single-spa-layout'));

                var applications = singleSpaLayout.constructApplications({
                    routes,
                    loadApp({ name }) {
                        return System.import(name);
                    },
                });

                var layoutEngine = singleSpaLayout.constructLayoutEngine({ routes, applications });

                applications.forEach(singleSpa.registerApplication);
                singleSpa.start();
            });
        });
    </script>
</body>
</html>

Hi @NightkingCH, thanks for the clear bug report. This does appear to be broken. Would you be interested in submitting a pull request with the fix?

Hi @joeldenning thanks for your fast response. Sure I can do that, I'll have a look into this.

Created a pull request for this issue: #54

This was resolved in #54

This was resolved in #54

Just a quick note that I am having this issue again where I have a div that I want to use as the containerEl but regardless if I pass the query selector or the actual element, all my micro frontends are appended at the bottom of body.
I am using json layout if that helps. And v1.5.0

@reggaemuffin please provide a demonstration of the issue, and open a new issue rather than commenting on this one. It's possible the fix didn't work for json layouts

Will try to get a minimal example going that reproduces this some time this week, but wanted to first alert you of the problem in case you already know about it :)

👍🏿  sounds good


Add XDM

Project URL
https://github.com/subhra74/xdm
Category
Downloader
Description
Add Extream Download Manager (XDM) to applications.json
Why it should be included to Awesome macOS open source applications  (optional)
Xtreme Download Manager (XDM) is a powerful tool to increase download speeds up to 500%, save streaming videos from YouTube, DailyMotion, Facebook, Vimeo, Google Video and 1000+ other websites, resume broken/dead downloads, schedule and convert downloads.
Checklist

[x] Edit applications.json instead of README.md.
[x] Only one project/change is in this pull request
[ ] Screenshots(s) added if any
[x] Has a commit from less than 2 years ago
[x] Has a clear README in English

          1 Warning

:warning:
Found 1 link issue, a project collaborator will take care of these, thanks :)

Link issue by awesome_bot

Line
Status
Link

8176
Dupe
https://github.com/subhra74/xdm

  Generated by :no_entry_sign: Danger

@kaconk17 Thank you! 👍


Mark DependencyArtifactDownloadProgressEventCrossVersionTest as flaky

@bot-gradle test and merge

I've triggered a build for you.


Wrong loader

Hi, thank you for your amazing work. I found your code always uses image_02 even if the side is 'r':
https://github.com/kieran514/robustdepth/blob/8d96d0e6dc3c69bcf21a211bb0c4414304dded96/Robust-Depth/datasets/kitti_dataset.py#L73-L83

Hello, thank you for your interest in our paper.
Our code only generates augmented versions of the left side of the images. Originally, it was designed to use both sides, but we choose to focus solely on the left side for the sake of simplifying augmentation creation. It's worth noting that this choice results in images being repeated, considering both left and right sides. However, this aligns with the number of images used per epoch in comparison to other methods.
One could argue that this approach leads to less unique data being processed by our method, potentially resulting in improved performance if both sides were used, particularly if right-side augmentations are created. Therefore, for best practice, we should consider incorporating right-side augmentations if they are created, as this may offer opportunities for future enhancements.
But this is a very valid point!


Fix exec_replace

Fix #125 and #129 (👀 maybe?)

👋
On the topic of our offline conversation about ctrl_c test on windows 🪟 :
The error you have been seeing seems to come from the subcommand spawn.
This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher. (os error 216)

It seems that the script definition used there is just not runnable on windows. (https://github.com/software-mansion/scarb/blob/f8494ed3290c5ae45aa7015a2cf36f38c12f58ff/scarb/tests/subcommand.rs#L144)
After changing the script definition, I was able to run it successfully and ensure it has been killed by ctrl_c.
To test this, I have created a main.rs file in scarb-test-support crate with following content: https://gist.github.com/maciektr/6b6c18a358b5792912bc0d0261ef3ac2
Then I have used this in tests instead.
    let addr = {
        let addr = listener.local_addr().unwrap();
        addr.to_string()
    };

    let mut child = Scarb::new()
        .std()
        .arg("test-support")
        .arg("hang-on-tcp")
        .arg("--address")
        .arg(addr)

I have not been able to make this check work though. I do not understand what is the purpose of this.
    match sock.read(&mut [0; 10]) {
        Ok(n) => assert_eq!(n, 0),
        Err(e) => assert_eq!(e.kind(), io::ErrorKind::ConnectionReset),
    }

I know it has been copied from Cargo verbatim. Maybe it is enough for now to only check if the process has been killed? cc @mkaput

I have not been able to make this check work though. I do not understand what is the purpose of this.
    match sock.read(&mut [0; 10]) {
        Ok(n) => assert_eq!(n, 0),
        Err(e) => assert_eq!(e.kind(), io::ErrorKind::ConnectionReset),
    }

I know it has been copied from Cargo verbatim. Maybe it is enough for now to only check if the process has been killed? cc @mkaput

This check tests if the socket either closed successfully without any unread rubbish left in the buffer, or died by the second party being terminated.

It seems that the script definition used there is just not runnable on windows. (
https://github.com/software-mansion/scarb/blob/f8494ed3290c5ae45aa7015a2cf36f38c12f58ff/scarb/tests/subcommand.rs#L144
)
After changing the script definition, I was able to run it successfully and ensure it has been killed by ctrl_c. To test this, I have created a main.rs file in scarb-test-support crate with following content: https://gist.github.com/maciektr/6b6c18a358b5792912bc0d0261ef3ac2 Then I have used this in tests instead.
    let addr = {
        let addr = listener.local_addr().unwrap();
        addr.to_string()
    };

    let mut child = Scarb::new()
        .std()
        .arg("test-support")
        .arg("hang-on-tcp")
        .arg("--address")
        .arg(addr)

May not be the cleanest flow, but imho worth considering.

I like this! I'd only rather use Command::new(cargo_bin!("scarb-test-support")) instead of Scarb::new().std(), seems more clear.


[WEBSITE] Make /docs/latest point 0.8.2

What is this PR for?
Update .htaccess to make /docs/latest point 0.8.2 version doc
What type of PR is it?
Improvement
Questions:

Does the licenses files need update? no
Is there breaking changes for older versions? no
Does this needs documentation? no

LGTM


fixing bugs related to otuput and improving speed

I calculated it and it is maximum on this situation, because MD is not mentioned as a character.

You can try it, if you want.

have you tested it?

Yes, of course. It works 100%.


Preset landmarks

As a designer, it would be convenient to have the commonly used landmarks relative to a cube pre-defined:

left
right
top
bottom
front
back
leftTop
leftBottom
leftFront
leftBack
rightTop
rightBottom
rightFront
rightBack
topFront
topBack
bottomFront
bottomBack
leftTopFront
leftTopBack
leftBottomFront
leftBottomBack
rightTopFront
rightTopBack
rightBottomFront
rightBottomBack

Resolved in #84


[proposal] PBxplore modularization

The current version of PBxplore is mostly oriented toward a use through the command line. With some refactoring, the program could also be used as a python module which would dramatically increase its extensibility. I hereby propose a structure for PBxplore as a python library in the hope it will lead to a conversation.
PBxplore provides a structure file parser, a Protein Blocks assignment implementation, and some analysis tools. The PB assignment and the analysis can be accessed by the user through command line programs. Yet, they cannot be accessed directly within a python script. A basic use of PBxplore within python would be to get a PB sequence from a protein 3D structure to run custom analyses. To do this with the current version of PBxplore, one needs to run PBassign.py using subprocess and read the fast file that is generated. Such procedure is burdensome; it implies to change context from python to the command line, to create a separate process, and to create a temporary file which should be deleted later. The corresponding code could be:
# Not tested of course, take with caution.
import subprocess
import os
import PBlib

# The name of the input file without extension
structure_name = ’test’
# The file name of the temporary fast file
fasta_fname = structure_name + ‘.PB.fasta’
# We call PBassign which generate a fast file
subprocess.call([‘PBassign.py’, ‘-p’, structure_name, ‘-o’, structure_name])
# We read the fast file using a function provided by PBxplore
headers, sequences = PBlib.read_fasta(fasta_fname)
# We delete the temporary fast fila as it is not useful anymore
os.remove(fasta_fname)
# We finally get the PB sequence we wanted
pb_sequence = sequences[0]

With the refactoring I propose hereby, obtaining a PB sequence from a PDB file could be coded this way:
import pbxplore
import pbxplore.structure

# The path to the input structure file
structure_fname = ’test.pdb’
# We read a single structure from the PDB file
protein_structure = pbxplore.structure.read_pdb_single(structure_fname)
# We get the PB sequence; as PB assignment only needs dihedral angles,
# the assignment method input is a list of dihedrals
pb_sequence = pbxplore.assign(protein_structure.dihedrals())

# Assigning a PB sequence from a PDB is a frequent need, so we can have
# a shortcut
pb_sequence = pbxplore.assign_from_pdb(structure_fname)

The same gain in readability would be obtained for the current analyses. In addition, a workflow that would do several analyses would mutualise steps that would be repeated in the current state of the program.
The post-refactoring exemple also shows the separation of concern and the loose coupling the refactoring would bring. In this proposal, the pbxplore module is divided into four main submodules:

pbxplore.assignment contains the functions directly related to the assignment
pbxplore.structure contains the structure file parsers
pbxplore.analysis contains the analysis functions
pbxplore.vizualization contains functions for data visualisation

Two other submodules are also be included for convenient purpose:

pbxplore.data contains PB related data such as the PB definition and substitution matrices
pbxplore.fasta_utils contains function to read and write fast files

Finally, shortcuts could be access through the pbxplore main module.
A major point of this proposal is the loose coupling between the submodules. The assignment method should not be aware of the structure parser and only asks for a list of dihedral angles. Same wise, the parser is not aware what will be done with the structure but provides, one way or another, a list of dihedral angles. This loose coupling make easier the addition of new parser or the bridges to third party parser. Here are some exemple of data source that could be used by PBxplore users:

GRO file parser
interface with the mdtraj library
output of g_angles
amber or charmm trajectory parser
PDB parser aware of protein structure specificities and experimental tricks

The same loose coupling would allow the analysis function to obtain the PB sequences from alternative sources too:

database of PB sequences
alternative assignment tool
alternative structural alphabet
data altered by other analyses and filters

By outputting the results using the simplest data structures, the module would ease significantly the development of new analyses and visualisation.
The command line tools are an important part of PBxplore and should be maintained and extended. In this model, they are wrapper around the corresponding functions in the library which allows clearer code and better testing.
Here is the suggested architecture of the PBxplore library. The architecture of the command line tools remains unchanged. Note that only the functions of the “public API” are reported bellow. Note also that some functions may be missing, especially in the analysis and visualisation modules.

pbxplore.structure

pbxplore.structure.read_pdb_single
pbxplore.structure.read_pdb_multi
pbxplore.structure.read_mmcif_single
pbxplore.structure.read_mmcif_multi
pbxplore.structure.read_xtc

pbxplore.assignment

pbxplore.assignment.assign
pbxplore.assignment.assign_matrix

pbxplore.analysis

pbxplore.analysis.count
pbxplore.analysis.distribution_matrix
pbxplore.analysis.neq
pbxplore.analysis.hclust

pbxplore.vizualization

pbxplore.vizualization.weblogo

pbxplore.data

pbxplore.data.definition
pbxplore.data.subtitution

pbxplore.fasta_utils

pbxplore.fasta_utils.read_fasta
pbxplore.fasta_utils.write_fasta

pbxplore

pbxplore.assign -> pbxplore.assignment.assign
pbxplore.assign_matrix -> pbxplore.assignment.assign_matrix
pbxplore.assign_from_pdb
pbxplore.assign_from_mmcif
pbxplore.matrix_from_pdb
pbxplore.matrix_from_mmcif
pbxplore.matrix_from_xtc

Alternatively, parts of the code that rely on third party tools such as mdanalysis and weblogo could be separated to ease the gestion of the dependancies.
Here are the tasks related to this proposal:

[ ] Finish the regression tests for all command line tools. These large changes should not affect how the command line tools works (invocation syntax, so as input and output files should remain unchanged).
[ ] Make PBxplore installable (see issue #21)
[ ] Reorganise the code to fit the proposed structure
[ ] Make sure all functions from the public API are documented
[ ] Write a narrative documentation

Agree but I sugget we transform one program after another, starting with PBassign.py

I agree it is safer. I'll start by having everything PBassign does available as a library.

Is everything ok ?
Sincerely

On 22/04/15 10:21, Alexandre G. de Brevern wrote:

Is everything ok ?
Sincerely
—
Reply to this email directly or view it on GitHub
https://github.com/pierrepo/PBxplore/issues/25#issuecomment-95071271.

6 pull requests merged yesterday. We are definitively progressing on the
matter.

Yes. @jbarnoud had hard time in both modularizing PBxplore and testing newly created functions. PBcount.py is now modularized.

cool !!!


[dv] Add ROM-only chip-level test example

A few chip-level tests (e.g., chip_sw_flash_init) must run and
complete entirely in the ROM stage. This means that the conventional
mechanism for running chip-level tests at the flash (ROM_EXT) stage is
not suitable for such tests.
This commit overcomes this by adding a mechanism to build a chip-level
test to be run in ROM using the meson build system. (Conveniently, the
opentitan_functest rule already provides such a mechanism using the
test_in_rom rule option.) Additionally, this commit adds a plusarg to
the DV sim environment to enable the testbench to skip loading a flash
image for such tests, as control flow will never reach flash. Lastly, an
example ROM-stage chip-level test is added to verify the correctness of the
added infrastructure, and to provide a reference point for future
chip-level test development.
Signed-off-by: Timothy Trippel<EMAIL_ADDRESS>

@davebwil , this should allow you to complete the chip_sw_flash_init test, which must run at the ROM stage.

Interestingly enough, the example_rom_chip_level_test.c added in this PR passes in Verilator, but fails in DV sim with:
UVM_ERROR @<PHONE_NUMBER> ps: (uart_monitor.sv:179) uvm_test_top.env.m_uart_agent0.monitor [uvm_test_top.env.m_uart_agent0.monitor] Expect uart_tx stable from 25 to 75 of the period, but it's changed

Any thoughts @sriyerg, @weicaiyang, @rswarbrick ? No worries if not, I will dig into it more.

Interestingly enough, the example_rom_chip_level_test.c added in this PR passes in Verilator, but fails in DV sim with:
UVM_ERROR @<PHONE_NUMBER> ps: (uart_monitor.sv:179) uvm_test_top.env.m_uart_agent0.monitor [uvm_test_top.env.m_uart_agent0.monitor] Expect uart_tx stable from 25 to 75 of the period, but it's changed

Any thoughts @sriyerg, @weicaiyang, @rswarbrick ? No worries if not, I will dig into it more.

What baud rate do you plan to run? The baud rate in design now is 1.4m. The TB is using 1mhz (the 2nd signal in the waves is TB's uart clock ).

Interestingly enough, the example_rom_chip_level_test.c added in this PR passes in Verilator, but fails in DV sim with:
UVM_ERROR @<PHONE_NUMBER> ps: (uart_monitor.sv:179) uvm_test_top.env.m_uart_agent0.monitor [uvm_test_top.env.m_uart_agent0.monitor] Expect uart_tx stable from 25 to 75 of the period, but it's changed

Any thoughts @sriyerg, @weicaiyang, @rswarbrick ? No worries if not, I will dig into it more.

What baud rate do you plan to run? The baud rate in design now is 1.4m. The TB is using 1mhz (the 2nd signal in the waves is TB's uart clock ).

Thanks for looking into this @weicaiyang !  I was just using the default baud rate defined in the device_sim_dv.c file in the test, which is the same used in the test ROM, which is 1m. Did the default design baud rate change? If I am not mistaken the docs say the baud rate is programmable only to 1m, is this incorrect?

Sorry I misread what you said. Anywho it looks like the period of cio_tx_o is around 700khz not 1.4mhz? and the TB expects it to be 1mhz (which is what it should get configured to in the example ROM test I added in this PR (L28 in sw/device/tests/example_rom_chip_level_test.c), since the UART configuration was copied directly from the test ROM, which initializes the UART baud rate to 1mhz). So something is not quite right.

Sorry I misread what you said. Anywho it looks like the freq of cio_tx_o is around 700khz not 1.4mhz? and the TB expects it to be 1mhz (which is what it should get configured to in the example ROM test I added in this PR (L28 in sw/device/tests/example_rom_chip_level_test.c), since the UART configuration was copied directly from the test ROM, which initializes the UART baud rate to 1mhz). So something is not quite right.

It's 1.4mhz in design. The data lasts for an entire cycle.
I check the waveform. tb.dut.top_earlgrey.u_uart0.u_reg.ctrl_nco_qs[15:0] is 0xebed, but for 1mhz, it should be 0xaaaa, based on this formula - baud_rate * (2**(nco_width+4)) / clk_freq  and nco_width is 16, clk_freq is 24mhz.
can you check if we get the right value for 2 constant - kUartBaudrate and kClockFreqPeripheralHz, which should be 1 mhz and 24 mhz?
      dif_uart_configure(&uart0, (dif_uart_config_t){
                                     .baudrate = kUartBaudrate,
                                     .clk_freq_hz = kClockFreqPeripheralHz,
                                     .parity_enable = kDifToggleDisabled,
                                     .parity = kDifUartParityEven,
                                 }));

The test runs successfully in DV and Verilator sims now.

The test runs successfully in DV and Verilator sims now.

Awesome, thanks for the update.

Oh sorry I was reading the data line as the clk line.... but alas I have fixed it thanks to your help @weicaiyang ! I had linked the wrong device file so the baud rate was indeed wrong for DV sim.

Thanks @weicaiyang!


msal-browser throws error code endpoints_resolution_error/openid_config_error

Core Library
@azure/msal-browser
Core Library Version
2.12.1
Wrapper Library
Not Applicable
Wrapper Library Version
n/a
Description
My app's error telemetry is showing MSAL's endpoints_resolution_error is the most common error for our users. From what I can tell MSAL throws this error if it can't load metadata during initialization. I'm unable to repro the error myself so I'm interested to hear from your team what might cause this error for users and if there's any recommended fix or debug steps.
A couple other data points from the telemetry data - The error only occurs on Chrome running Windows 10. I only see this error occurring in our production environment yet our int environment uses the same MSAL configuration.
Here's the error details:
 {
    "errorCode": "endpoints_resolution_error",
    "errorMessage": "Error: could not resolve endpoints. Please check network and try again. Detail: ClientAuthError: openid_config_error: Could not retrieve endpoints. Check your authority and verify the .well-known/openid-configuration endpoint returns the required endpoints. Attempted to retrieve endpoints from: https://login.microsoftonline.com/common/v2.0/.well-known/openid-configuration",
    "subError": "",
    "name": "ClientAuthError",
    "message": "endpoints_resolution_error: Error: could not resolve endpoints. Please check network and try again. Detail: ClientAuthError: openid_config_error: Could not retrieve endpoints. Check your authority and verify the .well-known/openid-configuration endpoint returns the required endpoints. Attempted to retrieve endpoints from: https://login.microsoftonline.com/common/v2.0/.well-known/openid-configuration",
    "stack": "ClientAuthError: endpoints_resolution_error: Error: could not resolve endpoints. Please check network and try again. Detail: ClientAuthError: openid_config_error: Could not retrieve endpoints. Check your authority and verify the .well-known/openid-configuration endpoint returns the required endpoints. Attempted to retrieve endpoints from: https://login.microsoftonline.com/common/v2.0/.well-known/openid-configuration\n at t [as constructor] (https://pi.dynamics.com/static/js/1b75e6aa3e0261f15c89.js:2:9024)\n at new t (https://pi.dynamics.com/static/js/1b75e6aa3e0261f15c89.js:2:15326)\n at Function.t.createEndpointDiscoveryIncompleteError (https://pi.dynamics.com/static/js/1b75e6aa3e0261f15c89.js:2:15856)\n at Function.<anonymous> (https://pi.dynamics.com/static/js/1b75e6aa3e0261f15c89.js:2:99917)\n at https://pi.dynamics.com/static/js/1b75e6aa3e0261f15c89.js:2:4216\n at Object.throw (https://pi.dynamics.com/static/js/1b75e6aa3e0261f15c89.js:2:4321)\n at s (https://pi.dynamics.com/static/js/1b75e6aa3e0261f15c89.js:2:3110)"
}

Incase it's helpful, we also saw this error code when running v2.9.0 and the error was this:
{
    "errorCode": "endpoints_resolution_error",
    "errorMessage": "Error: could not resolve endpoints. Please check network and try again. Detail: SyntaxError: Unexpected end of JSON input",
    "subError": "",
    "name": "ClientAuthError",
    "message": "endpoints_resolution_error: Error: could not resolve endpoints. Please check network and try again. Detail: SyntaxError: Unexpected end of JSON input",
    "stack": "ClientAuthError: endpoints_resolution_error: Error: could not resolve endpoints. Please check network and try again. Detail: SyntaxError: Unexpected end of JSON input\n at t [as constructor] (https://pi.dynamics.com/static/js/3030e6970db3ac7895af.js:2:8843)\n at new t (https://pi.dynamics.com/static/js/3030e6970db3ac7895af.js:2:14857)\n at Function.t.createEndpointDiscoveryIncompleteError (https://pi.dynamics.com/static/js/3030e6970db3ac7895af.js:2:15387)\n at Function.<anonymous> (https://pi.dynamics.com/static/js/3030e6970db3ac7895af.js:2:94906)\n at https://pi.dynamics.com/static/js/3030e6970db3ac7895af.js:2:4161\n at Object.throw (https://pi.dynamics.com/static/js/3030e6970db3ac7895af.js:2:4266)\n at s (https://pi.dynamics.com/static/js/3030e6970db3ac7895af.js:2:3055)"
}

MSAL Configuration
{
    auth: {
      authority: 'https://login.microsoftonline.com/common',
      clientId: process.env.CLIENT_ID,
      navigateToLoginRequestUrl: false,
      redirectUri: window.location.origin,
      postLogoutRedirectUri: null,
    },
    cache: {
      cacheLocation: 'localStorage',
    },
    system: {
      loggerOptions: {
        piiLoggingEnabled: process.env.NODE_ENV === 'development',
      },
    },
  }

Relevant Code Snippets
No response
Identity Provider
Azure AD / MSA
Source
Internal (Microsoft)

@deasems A simple mitigation you can try is to retry a request that gets this error (just retry once). I would be interested to know how often retrying fixes it. In lieu of a repro, correlation IDs of failed requests might be helpful for us to determine what went wrong.

Thanks for the tip @jasonnutter, we'll work on adding retry logic and check the success rate.
Is it possible to programmatically get correlation IDs for failed MSAL requests so we can send those in our telemetry? Currently we're logging the error object that MSAL provides which doesn't include a correlation ID.

@deasems You can generate your own correlation id and pass it into the request.

@jasonnutter will that correlation ID be included in the error object for endpoints_resolution_error or would I need to keep track of the ID used for each request to send it to my telemetry if the request fails.

@deasems We can add that I suppose, but under a conditional to not expose it by default to all users, in case some folks prefer otherwise. We have some work pending on correlation ids this quarter, we will consider this during the design.

@jasonnutter will that correlation ID be included in the error object for endpoints_resolution_error or would I need to keep track of the ID used for each request to send it to my telemetry if the request fails.

You will need to keep track of it.


karma: ReferenceError: Can't find variable: require

When I am running my qunit tests using karma I am getting this error when loading localforage library:
ReferenceError: Can't find variable: require
The issue is that qunit defines "module" and "module.exports" so this check we have passes (but require is undefined)
 if (typeof module !== 'undefined' && module.exports)

I have a fix for it, changing it to:
if (typeof module !== 'undefined' && module.exports && typeof require !== 'undefined')

fixes the issue. There might be a different fix when I set up karma or qunit but I didn't find one, and it might be good to check for require before using it.
I will submit a pull request for this bug in a few minutes

Is this pull request getting any movement?  We found several copies of this line in localforage.js that need the modification in order to run under karma.

I don't actually see a related pull request. If you could submit the code/PR that'd be great; I can take a look soon :smile:

I's here:
https://github.com/mozilla/localForage/pull/366

Oops, sorry, I’m dumb. Looking now!
-tofumatt
On 13 May 2015 at 17:07:59, Mike<EMAIL_ADDRESS>wrote:
I's here:
#366
—
Reply to this email directly or view it on GitHub.

Fixed in #378.