Create bytecode compiler and evaluation

Hi,
This is a first pass at creating a bytecode implementation for the language. I noticed you (offhand) discussed making a bytecode in comments on another issue, so I'm hoping that you will be interested in merging this once it's more complete. Definitely feel free to let me know if you'd rather parts of this were designed or implemented differently, or if you don't feel that it's an appropriate feature in the first place.
This isn't done, but it's at the point where I'm pretty sure I will finish it, and it's beginning to be at the point where I don't want to be accidentally duplicating work. Right now cargo test --features="no_object no_index" -- --skip eval --skip test_timestamp --skip test_type_of passes and it hasn't been optimized at all.
My eventual goal for this is to use it in a game I'm writing to let players script their units actions. I'm hoping that the bytecode will improve performance when repeatedly executing the same script, give me a deterministic measure of work done by counting instructions executed, and let me implement stuff where user scripts can act as a generator that yields values and pauses/resumes.
So far none of those goals have been achieved, however I think there is a pretty clear path forwards from this implementation to all of them. I'm also not sure if the parts needed to achieve the second two goals will really belong in the core language, but they could live behind feature flags, or I could maintain a fork, since the amount of code needed for them should be pretty small.
TODO - In the immediate future

[ ] Set up testing properly, ideally so that tests test both executors and compare the output.
[ ] Implement objects, index, in, and type_of.
[ ] Maybe implement eval. How attached are you to this function? Another option might be to have any function that calls eval to execute as an ast instead.
[ ] Keep track of Positions. Possibly with a second source map array in the bytecode struct.
[ ] Set up benchmarks

TODO - But it could probably wait until after this is merged

[ ] Broaden the API to match the one for ASTs, e.g. by calling it with an existing Scope, calling arbitrary functions, merging multiple pieces of bytecode, and so on.
[ ] Improve efficiency of bytecode representation

[ ] Make each individual variant smaller (e.g. by interning Strings)
[ ] Store bytecode as an array of bytes, so that small instructions can take up less space than large ones. Or maybe make clever use of unions to do the same.

[ ] Improve efficiency of executor (I'm maintaining multiple stacks, since it was the easiest way to implement things, not the most performant. I haven't done any work on optimizing the function).
[ ] Improve efficiency of foreign function calls (This is probably going to be hard. Currently we search for functions at run-time because it depends on the types of the arguments. One option might be to add a type inference phase, then whenever we managed to determine the types we could directly specify the function in the bytecode).

TODO - Potential future projects

[ ] Maybe fuzz the two executors for differences
[ ] JIT bytecode to asm?

TODO - But not in this pull request and maybe not in this repository

[ ] Some form of Yield(Dynamic, State) return option that allows for storing the execution state and restarting at well.
[ ] Instruction counting in the executor, possibly weighting instructions with some form of approximate execution cost.
[ ] The option to pause the executor after so many instructions are executed (similar to Yield but without a value).

This is great.  You're probably referring to https://github.com/jonathandturner/rhai/issues/100 where I think compiling Rhai down to a bytecode format with an interpreter may make it run faster for the really demanding applications (e.g. games).
Can I suggest you base your changes on https://github.com/schungx/rhai/tree/bytecodes where I do most of the on-going work to avoid merge conflicts later.  That branch bytecodes is newly created so you can base your PR's.

Do you plan on retaining the AST as a valid execution format and generate bytecodes from the AST, or compile directly to bytecodes?

Maybe implement eval. How attached are you to this function? Another option might be to have any function that calls eval to execute as an ast instead.

Not attached to it at all.  Just put it in because it was dead simple in the beginning.  Now I realize why people who work on JS optimizers curse the eval.
Since an eval script is ever only run once each time after it is generated, a good strategy is to defer to AST-walking.  You still need to have an efficient strategy to deal with the additional variables in the scope throwing off your offsets though...

Can I suggest you base your changes on https://github.com/schungx/rhai/tree/bytecodes

Sure, will do.
Have you considered making schungx/rhai the main repo for the language?

Do you plan on retaining the AST as a valid execution format and generate bytecodes from the AST, or compile directly to bytecodes?

For my purposes I'll be compiling directly to bytecode. The ability to implement the aforementioned features is necessary, and I don't see how I could do that easily with an AST, not to mention that performance should be better.
If you're asking from a language perspective I assume AST compilation will stick around. At least for one off scripts without loops I think it should be faster. It's certainly too early to think about killing it.

Since an eval script is ever only run once each time after it is generated, a good strategy is to defer to AST-walking. You still need to have an efficient strategy to deal with the additional variables in the scope throwing off your offsets though...

The idea is to just leave the entire function containing the eval in the ast-executor code. Calling an ast function from bytecode should be no harder than calling any other foreign function.  That should drop all the extra variables before returning to bytecode.

Do you have any preliminary benchmarks?

I just quickly set up a version of the fibonacci benchmark in response and out of curiosity.
test bench_iterations_fibonacci          ... bench: 286,836,385 ns/iter (+/- 2,348,435)
test bench_iterations_fibonacci_bytecode ... bench: 265,362,002 ns/iter (+/- 801,004)

Considering that no effort has gone into optimizing it, and that there are some easy looking wins, I'm pretty happy with it being faster at all.

Yes, I've been thinking about it also. Profiling runs with Rhai show that function-name/arg-types hashing consistently shows up on top of IR counts. Therefore, dispatching to the correct function appears to be a hot-path bottleneck, although so far I haven't thought of a way to do it faster than hashing everything.

There are probably very few overrides of any given function on average, and small linear searches are typically faster than hashes, so something like this might work well to speed up the search (note that the hot path doesn't touch a String at all):
struct FunctionsTable {
     function_names: HashMap<String, usize>,
     functions: Vec<Vec<FnType>>,
}

fn register_fn(table: &mut FunctionsTable, fn_name: String, fn: FnType) {
    if let Some(idx) = table.function_names.get(&fn_name) {
        table.functions[idx].push(fn)
    } else {
        table.insert(name, functions.len());
        functions.push(vec![fn]);
    }
}

enum ExprOrBytecode {
    Call{ fn_name_idx: usize, args: Vec<Dynamic> }
}

fn exec_call(table: &mut FunctionsTable, fn_name_idx: usize, args: Vec<Dynamic>) {
    let functions = table.functions[fn_name_idx];
    for function in functions {
         if accepts(function, args) {
             return function(args)
         }
    }
    return error
}

Have you considered making schungx/rhai the main repo for the language?

Not really.  I am mostly only tinkering based on the fabulous work of people before me.

Compiling directly to bytecode will probably be a large surgery on the parser. Optimizing machine-generated scripts with spliced-in constants also works more easily on an AST than on bytecodes.
For your purposes, I'd assume that you would be compiling your users' scripts into bytecodes, and then persisting the bytecode stream so it can be reloaded later and no need to re-parse, so it might be more useful to keep the AST around.

Ah, sorry, we're talking about different things.
I don't particularly have a problem with compiling "source -> AST -> bytecode", maybe someday I will want to optimize that but for now it seems fine. I just don't expect to ever actually execute the AST.
I'll probably persist the original source (for display and deduplication reasons) and persist/cache the bytecode... As long as I keep a version of the bytecode around I don't think I'll need the AST to reload it, but if it ends up being useful it's not an issue to store it as well.

That would then change the semantics of the language because you're allowed to introduce new variables via eval. You can forbid it in bytecodes though, but then probably the engine should raise a runtime error when an eval script runs a let in global scope instead of silently discarding the new variable.

Am I mistaken in thinking that eval inside a function can't introduce variables outside that function? Top level eval would mean backing out the top level "function" to the AST version just like everywhere else, but functions that don't themselves contain eval should still be fine?

Not really. If you think about it, a search will have to compare all the TypeId's of all the arguments, as well as the function name, for each single function entry. There are a lot of functions in the standard library (around 300 for the full library). All these comparisons are probably going to be slower than just hashing it once.

Check out the code snippet, we skip comparing function names and only have to compare TypeId's for functions with the same name.

In the pathological case of multiple overloading (e.g. arithmetic functions supporting all integer types), then it will be slower.

True, and these are really common operators...
Thinking more it might be possible to have our cake and eat it too, at ast/bytecode-build-time decide whether to go the hash route or the lookup route depending on the number of overloads.

Am I mistaken in thinking that eval inside a function can't introduce variables outside that function? Top level eval would mean backing out the top level "function" to the AST version just like everywhere else, but functions that don't themselves contain eval should still be fine?

That's the point.  eval is essentially equivalent to running the script code in-place, including defining new variables.  Except that functions cannot be defined in eval.
At top level, eval will introduce new variables.  At block level (such as a {} block or inside a function), any new variables introduced will be local.  That's why always_search is reset to false at the end of a block.

(note that the hot path doesn't touch a String at all):

Yes it does. You need to map a function name to its index. It can be pre-mapped if using only one AST, but when using multiple AST's plus user-loaded packages, you cannot predict the index of a function based on its name alone. In other words, you'll need to do a table lookup based on the name string. Of course, compiling to bytecodes assumes you won't add new functions later on, so you can freely do this optimization. Unless you want to give your users freedom to add/mix functions in bytecodes form, then you'll need to do the name lookups.
One way to handle it is to cache the name->index mapping and only incur this cost once, until the functions library is changed. Then the cache is invalidated.

Hmm, yes, it does require some sort of linking step that is going to take time O(code_size). That's probably a reasonable thing to do in an AST optimization pass though. The bytecode version already has a pass like this called resolve_fn_calls used to optimize bytecode->bytecode calls.

test bench_iterations_fibonacci ... bench: 286,836,385 ns/iter (+/- 2,348,435) test bench_iterations_fibonacci_bytecode ... bench: 265,362,002 ns/iter (+/- 801,004)

And Fibonacci is probably not a good benchmark to test for your use case as it is almost exclusively recursion.  The workload will be on stack management which you probably can't save much.
You'll want to use speed_test or primes (if you have arrays) to benchmark calculation and loop-heavy scenarios.

@gmorenz  Unfortunately I added modules support so you're going to have to deal with functions and variables qualified by modules (and possibly a way to load modules that are pre-compiled to bytecodes).
I can help out with some of the infrastructure and plumbing after you get bytecodes to work without modules -- meaning it compiles under no_module -- that would probably be enough.

@gmorenz Unfortunately I added modules support so you're going to have to deal with functions and variables qualified by modules (and possibly a way to load modules that are pre-compiled to bytecodes).

Ya, that's no problem. Some form of modules were on my wants for this language anyways, and I definitely already have ideas about how to implement modules and linking of modules.
Current work here is still on creating a clean and efficient index chain model... it turns out that the semantics don't fit a stack machine particularly cleanly. It's close to done, but I got sidetracked on the get_mut idea.

Current work here is still on creating a clean and efficient index chain model... it turns out that the semantics don't fit a stack machine particularly cleanly. It's close to done, but I got sidetracked on the get_mut idea.

Yes, array indexing is hairy especially when the language doesn't have pointers... otherwise it is a piece of cake, but then you'll have to probably throw in a GC...

As mentioned in schungx/11 I do not anticipate making time to finish this work anytime soon. Closing this for now. Feel free to re-use any part of this in any way if you so desire.
Thanks for the warm welcome @schungx.


Alerter doesn't recognise global.properties for alert.type

Custom alerters can not be set for all Azkaban Flows in global.properties or even uploaded flow properties files.
alert.type is specifically checked from the UI flow parameters only. It should check all possible configuration locations.

Hi Azkaban Team,
I noticed nobody was assigned to it, but if there’s already a solution in progress I’m happy to try helping out elsewhere.
Besides we've experienced the same issue, so this fix would be helpful
Approach is straightforward - read gloabal.properties into some final globalProps at ExecutorManager, and use it in case if options.getFlowParameters() doesn't contain key alert.type
Thanks!

The fix https://github.com/azkaban/azkaban/pull/1727

What is different between 'flowParameters' and 'flow.props'.
in my mind, flowParameter should including the flow.props(global.properties).
but it's not.


Replace old repo URL with GitHub repo URL

Replace old repo URL with GitHub repo URL
Summary
Pretty much self-explanatory.
Description
Replace the old project repo URL with the GitHub repo URL in the following files:

lib.rs
README.md

update MIT license URL
update CI pipeline badge

@just-a-stream pls don't merge after approval, just acknowledge it and then we'll chat about the best / optimal merge strategy offered by GitHub PRs (Create a merge commit vs Squash and merge vs Rebase and merge).
I personally recommend Rebase and merge for these tiny change PRs...
And maybe Squash and merge for larger PRs, given that they have a proper commit message with body written... Alternatively those could be merged with a default merge commit. That would have an advantage of atomic commits & nice incremental overview of the changes applied, but creates an additional potentially-redundant commit. Also, in that case no merge commits inside the feature branch itself should be allowed (outdated changes should be rebased insted).

The change LGTM


Create Service View

Service View must show service details and be fully functional.

Client Service views and fragment is created.


Add GF check: Format DESCRIPTION files HTML

Observed behaviour
I just rewrote the DESCRIPTION file for Comic Neue (https://github.com/google/fonts/pull/2405/files) and I reformatted the HTML to have

one line per structure tag
one line per sentence
a final blank line
no mid-sentence line breaks

Expected behaviour
We already have checks for DESCRIPTION files, and it would be good to have a WARN check for these 'bike shed' structure; it doesn't make a difference to rendered output, but the benefit of this approach is that it makes subsequent diffs very clean.

In a private discussion, it came up which tags are currently rendered on  fonts.google.com
Yes:
No:

 and similars would be also a No, AFAIK

Yes

No:

 are now allowed!
https://github.com/googlefonts/gf-docs/tree/master/Spec#upstream-repo-structure now lists the html tags allowed in our description files.
Inline tags:
a, em, i, strong, b

Block tags:
p ul ol li

So, we should have a few other FB checks, that DESC files:

[ ] Do have closing tags for all elements
[ ] Do not contain any other tags
[ ] Do not have any attributes
[ ] Block tags are on their own line
[ ] Inline tags are not on their own line

Related issues:
https://github.com/googlefonts/fontbakery/issues/2550
https://github.com/googlefonts/fontbakery/issues/2937
https://github.com/googlefonts/fontbakery/issues/2671
https://github.com/googlefonts/fontbakery/issues/2667

thanks! This is a very detailed description of the supported tags. I'll use it to implement the checking routines.

For general HTML conformity:

[ ] Inline tags do not contain block tags
[ ]
 and
 must only contain only
 tags and white space for line breaks and indentation

I don't really get these though, is it just a style guide for the markup? As for the displayed result these rules are irrelevant:

[ ] Block tags are on their own line
[ ] Inline tags are not on their own line

And how about this?:

[ ] All tags must have content (tags or text content), that content is not only white space. E.g. invalid:     or  or
 \n <\p>

Oh, and this is too vague:

So, we should have a few other FB checks, that DESC files:

[ ] […]
[ ]  Do not have any attributes

Because, a "DESC files" can't have HTML attributes and if it is for all tags:  needs the href attribute to be whitelisted.

@felipesanches since we are going to be updating a lot of the descriptions this year, I'd like to up the priority of this check

In https://github.com/google/fonts/commit/a57e5a286ba6c184dd528255b01dd097a0adb515 I removed the https://www. from the text inside an ; this check should FAIL descriptions with that or https:// so that the URLs look nice

At some point I started drafting an implementation for this, and left it unfinished at an open draft-PR (#3438). I am closing that now and leaving the wip patch here in case we give it another try in the future.
issue_2811_wip.txt
Before putting more time into this, I'd like to know whether these HTML formatting requirements are still needed.

Apart from what Dave lists in the original comment of this issue, you can find more context on the requirements in the Description section of the GF-Guide. It includes the allowed HTML elements for both the DESCRIPTION and the ARTICLE html files.

I confirm the forbidden elements here, https://github.com/googlefonts/googlefonts.github.io/pull/128/files
The rationale for checking for forbidden elements is that sometimes they have crept in, and then halted the push process; this is very annoying because it is something of a triviality given a lot of work goes into preparing the font files themselves, and it all gets delayed just because of a stray HTML element...

@felipesanches could we have a forbidden html elements check that raises a FATAL error code please?

@m4rc1e yes!


fixed user guide link

I was confused that the user guide link did not direct to the actual user guide page. It was actually intended to link the installing and configuring guide. Thus, I changed the wording from 'user guide' to 'installing and configuring guide'. In addition, I added a sentence directing people to the 'Getting started' link. I would like to go to the 'Getting started' page from the home page.

Thanks for the fixes, I made two suggestions. I don't think internal rst links should have a file extension so this was always wrong.

Thank you!


feat: add baseline expectedPlayoutItems support to Playout Gateway

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
A feature and minor config manifest changes related to https://github.com/nrkno/tv-automation-state-timeline-resolver/pull/182

What is the current behavior? (You can also link to an open issue here)

What is the new behavior (if this is a feature change)?

Adds baseline expectedPlayoutItems support
Adds config for Viz Engine REST port config
Fixes config property name

Other information:

Status

[ ] Code documentation for the relevant parts in the code have been added/updated by the PR author
[x] The functionality has been tested by the PR author
[ ] The functionality has been tested by NRK

Codecov Report

Merging #521 (efb8b40) into release35 (799874b) will increase coverage by 14.46%.
The diff coverage is n/a.

@@              Coverage Diff               @@
##           release35     #521       +/-   ##
==============================================
+ Coverage      67.82%   82.28%   +14.46%
==============================================
  Files            251       12      -239
  Lines          15107      858    -14249
  Branches        3538      197     -3341
==============================================
- Hits           10246      706     -9540
+ Misses          4693      150     -4543
+ Partials         168        2      -166

Impacted Files
Coverage Δ

meteor/lib/collections/MediaWorkFlows.ts

meteor/lib/collections/PartInstances.ts

meteor/server/cronjobs.ts

meteor/lib/collections/MediaWorkFlowSteps.ts

meteor/server/api/ingest/genericDevice/actions.ts

...eor/server/api/ingest/syncChangesToPartInstance.ts

meteor/server/api/integration/media-scanner.ts

meteor/server/migration/1_7_0.ts

meteor/lib/api/pubsub.ts

meteor/server/api/ingest/cache.ts

... and 229 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 799874b...efb8b40. Read the comment docs.

This again? Sorry, VS Code doesn't indicate or warn about it when it automatically stages all changes.


Firebase functions integration

Hello, I'm still struggling with this, maybe you can help,

When I execute 'npm run build' no matter if I comment out the firebase import, it will trigger the following error that I googled and couldn't find any specific answer, the error will only dissapear if I comment out the geofirestore import...

package.json
"dependencies": {
"firebase": "^5.0.4",
"firebase-admin": "~5.12.1",
"firebase-functions": "^1.0.4",
"geofirestore": "^1.2.0"
},

I don't think this is a Geofirestore question as much as this is a Firebase Functions question. Either way, I'd tell you to do more research and reading the docs of firebase functions, because you dont need to import * as firebase from 'firebase'; And if you need to create a collectionref for geofirestore you could do...
import * as admin from 'firebase-admin';

const collectionRef = admin.firestore().collection('collection');

I know that admin is the normal way to query collections un cloud functions however if you try to assign that collectionRef to a new Geofirestore(collectionRef), you will get a casting error as Geofirestore  "firebase.firestore.CollectionReference", firebase-admin's collection type "FirebaseFirestore.CollectionReference", In another thread @invention7 sugested to import firebase package instead of admin, but as I show you before even importing Geofirestore as its simplest way throws that error at build.

I agree with @stubborncoder that firebase-admin's 'collection' type appears to be incompatible with the collection type expected by the geofirestore constructor.  To get around this, I created by collectionRef with the 'firebase' library.
Maybe, @MichaelSolati , there is an opportunity for Geofirestore in expanding the constructor to allow the firebase-admin's collection... but in terms of David's immediate need, I think it can be handled as-s.
@stubborncoder, in your firebase config, have you disabled persistence?
e.g.
PersistenceEnabled: false
?

Hi @invention7 my firebaseconfig wasn't modified at all, it's just as it's created by the firebase CLI... thanks for your help!

I'd be open to a PR if anyone wants to expand the types allowed. I'd also ask that the PR is made towards the dev branch.

Hey @stubborncoder, sorry I missed your comment...  try downgrading firebase to '5.0.0'

In general I can probably double type everything (it would just take a while to get through). You should also be able to // @ts-ignore type issues for now. While it's not the end all be all solution, it should work until I dual type everything.

@invention7  that did the trick m8, I downgraded as you suggested and now it builds and runs without any further issues!. Thank you very much!.

np @stubborncoder !  Good luck!!

oh wow!  awesome @MichaelSolati !

Thank you @invention7 !
For everyone else, Firebase Functions should work (may need to make a small patch, but for the most part it's good to go). Try npm i<EMAIL_ADDRESS>


Replace deadlink inside encoded_string_spec.rb

Thanks for pointing this out, I've amended the link back to ruby/spec as it's intended to be a link to a specific commit within the spec not ruby itself.


Fix container children drawing in overlapping containers

Describe the bug
When containers are overlapping, children from one container can draw over the foreground in the other container if the foreground in that container is empty.
Expected behavior
Containers should have full autonomy over their own draw region, and the empty background should not contain foreground content from components that are not children or children of siblings (in case sibling drawOrder is higher)
Versions (please complete the following information):

Imevul UI: 1.0.0
Cobalt 2.0.0b1
ComputerCraft CC:Tweaked

Fixed for next release


⚠ Carbubu has degraded performance

In 7f8cc30, Carbubu (https://carbubu.fr) experienced degraded performance:

HTTP code: 200
Response time: 8803 ms

Resolved: Carbubu performance has improved in 5f1cbf9 after 8 minutes.


Where is motion detect?

Hello! Where is motion detect future? Is it planned?

I am planning to integrate this feature. But it looks like i have to extend the extend the onvif-nvt modul, so it takes a while.

ok, thanks!


Rbac virtual attributes

Overview
When running queries, we are putting more and more virtual attributes into the main query. This transitions us from running N+1 queries from ruby to only running 1. Just running them as subqueries.
Running them as sub queries still runs each of the queries, but it lets us avoid downloading a ton of data to then do a COUNT, SUM, or FIRST. So if you look at the EXPLAIN plans, the queries are still happening, but in a much more controlled manner.
addresses

https://bugzilla.redhat.com/show_bug.cgi?id=1686433

Problem
This is a PR/BZ after all. There is a problem.
We are running these sub queries once for every row in the database (6k), not for every row returned (20). Typically, this is a lot faster and still less work than downloading all that data to ruby.
Well it is faster until the number of rows in the base table is sufficiently large, and the virtual attribute subquery is sufficiently slow.
And yes, we found that edge case for the Service explorer page.
Solution
The solution is to run the subquery once for every row on the screen rather than every row in the base table. To get this, we run the query with the WHERE and LIMIT as an inline view (subquery in the FROM clause). So the database is running for "every row in the base table", we just pretended like the 20 rows in the result set were "every row in the table."
Active record has the from() method to allow us to do this pretty easily.
-- before
SELECT "base".*,
       virtual_attribute /* 6k times */
FROM   "base"
WHERE  "base"."name" LIKE '%good stuff%'
LIMIT  20

-- after
SELECT *,
       virtual_attribute /* 20 times */
FROM (
    SELECT "base".*
    FROM   "base"
    WHERE  "base"."name" LIKE '%good stuff%'
    LIMIT  20
) AS "base"

WIP status: this works. I put in a kludge to enable this for every page possible.
In the short term, I would like us to test this across as many pages as possible, just to find issues.
then when we are ready to merge, I'll remove the hack and enable on specific pages

@NickLaMuro just fixed cops. All good with you now?

un WIP:
I was able to condense the change in search to a single block with 4 lines and a s/=/||=/.
Determining if we "should" optimize the query can be further simplified to just checking options[:use_sql_view]. I feel more comfortable with the more thorough check: It determines if we would benefit by optimizing the query, and checks some common ways it can blowup.
Converting the references() to left_joins is a bit more than I would have like. It iterates the includes (a symbol, string, array, or hash) and adds non polymorphic references. I'm sure there are some edge cases that can creep in here from other models, like a hash pointing to a polymorphic relationship, but we'll address that later when we want to introduce more generalized consumption.
I needed to use includes instead of references because we currently misuse references (we should be passing in an array and we pass in a hash, so it is just saying "require everything")

Trivia: turns out klass is not necessarily the same as klass.send(scope).klass - there is only 1 example where this fails, and it is in automate workflow where the scope name passed in changes the base class. Which I didn't think was legal but here we are.
So I changed the code to read scope.klass instead of using klass
I'm wondering if we want to call klass = scope.klass after we send(scope) in the few places.
We definitely have some duplicate code in the code that handles targets/klass/scope/array/...

@kbrock is this for hammer? I think so, but want to make sure.  Please add the labels if so. This change looks good overall.  Do you know what will be opt-ing in via use_sql_view?  What will use this new code for master/backport?  I believe we were going to make this the default in a followup PR for master but for backport to hammer, only targeted to one or two yamls like service.yaml.

fixed cops

@Fryguy Are you good with this now?

Hammer backport details:
$ git log -1
commit a688d0f993620553d633d59bc3e9983e1708adb3
Author: Jason Frey<EMAIL_ADDRESS>Date:   Fri Mar 22 13:24:24 2019 -0400

    Merge pull request #18543 from kbrock/rbac_virtual_attributes

    Rbac virtual attributes

    (cherry picked from commit 3dfb0b05be6645a67ebabc213bd544e0482834da)

    Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1688937

@kbrock Please take a look at Travis failure in hammer branch:
  1) Rbac::Filterer common setup with inline view handles added include from miq_expression
     Failure/Error: expect(results.first).to eq([service1])
       expected: [#<Service id:<PHONE_NUMBER>0074, name: "service_0000000000074", description: nil, guid: "e01e614e-8a06-...state: nil, retirement_requester: nil, tenant_id:<PHONE_NUMBER>0346, ancestry: nil, initiator: "user">]
            got: #<ActiveRecord::Relation []>
       (compared using ==)
       Diff:
       @@ -1,2 +1,2 @@
       -[#<Service id:<PHONE_NUMBER>0074, name: "service_0000000000074", description: nil, guid: "e01e614e-8a06-4262-a13c-9a89339ba208", type: nil, service_template_id:<PHONE_NUMBER>0016, options: {}, display: false, created_at: "2019-03-29 17:13:26", updated_at: "2019-03-29 17:13:26", evm_owner_id: nil, miq_group_id:<PHONE_NUMBER>0497, retired: false, retires_on: nil, retirement_warn: nil, retirement_last_warn: nil, retirement_state: nil, retirement_requester: nil, tenant_id:<PHONE_NUMBER>0346, ancestry: nil, initiator: "user">]
       +[]
     # ./spec/lib/rbac/filterer_spec.rb:2130:in `block (4 levels) in <top (required)>'


Tenant tokens

Is your feature request related to a problem? Please describe.
Support for tenant tokens according to the meili docs

Fill add instance form with tenant token?

No i meant managing tenant keys like the api keys but it seems that they are short-lived so there is no need to.
I will close the comment


fix some packages not being included, because of badly parsed folded field

according to https://www.debian.org/doc/debian-policy/ch-controlfields.html
Binary filed is a folded type.
During building boost genchanges folds Binary to multiple lines. (it has many binary packages from one source)
When aptly trims whitespace it then contatenates package names at the end of the line and beginning of the next line

@sobczyk thanks for discovering this bug! I've fixed it in a bit different way, which should be still compatible (whitespace is preserved)

Fix should be available as nightly build aptly_0.9.6~dev+145+g76bf7cb


Get authzed-dotnet up to feature parity with authzed-java

Seems like authzed-java will be a good reference.

[ ] Make sure that CI is generating things properly
[ ] Fix up buf.gen.yaml
[ ] Add client class that implements grpc-preshared-key

Add other TODOs as necessary.

I haven't finished #6 yet, but that shouldn't prevent users from using the lib as it currently stands, and I can do manual reruns in the meantime. I'm gonna go ahead and close this.


Microsoft.Network/virtualNetworks (2019-11-01)

Describe the noise
Resource type (i.e. Microsoft.Storage/storageAccounts)
Microsoft.Network/virtualNetworks
Microsoft.Network/virtualNetworks/subnets
apiVersion (i.e. 2019-04-01)
2019-11-01
Client (PowerShell, Azure CLI, or API)
PowerShell
Relevant ARM Template code (we only need the resource object for the above resourceType and apiVersion, but if it's easier you can include the entire template
 {
      "tags": {
        "displayName": "hub vnet"
      },
      "type": "Microsoft.Network/virtualNetworks",
      "name": "[parameters('hubVnetName')]",
      "apiVersion": "2019-11-01",
      "location": "[resourceGroup().location]",
      "properties": {
        "addressSpace": {
          "addressPrefixes": [
            "[parameters('hubVnetPrefix')]"
          ]
        },
        "enableDdosProtection": false,
        "enableVmProtection": false
      },
      "resources": [
        {
          "type": "subnets",
          "name": "[variables('mgmtSubnetName')]",
          "apiVersion": "2019-11-01",
          "properties": {
            "addressPrefix": "[parameters('mgmtSubnetPrefix')]",
            "networkSecurityGroup": {
              "id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('nsgName'))]"
            }
          },
          "dependsOn": [
            "[resourceId('Microsoft.Network/virtualNetworks', parameters('hubVnetName'))]",
            "[resourceId('Microsoft.Network/networkSecurityGroups', variables('nsgName'))]"
          ]
        },
        {
          "type": "subnets",
          "name": "[variables('sharedSubnetName')]",
          "apiVersion": "2019-11-01",
          "properties": {
            "addressPrefix": "[parameters('sharedSubnetPrefix')]",
            "networkSecurityGroup": {
              "id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('nsgName'))]"
            },
            "serviceEndpoints": [
              {
                "service": "Microsoft.ContainerRegistry"
              },
              {
                "service": "Microsoft.Web"
              },
              {
                "service": "Microsoft.EventHub"
              },
              {
                "service": "Microsoft.Storage"
              },
              {
                "service": "Microsoft.AzureActiveDirectory"
              },
              {
                "service": "Microsoft.AzureCosmosDB"
              },
              {
                "service": "Microsoft.KeyVault"
              },
              {
                "service": "Microsoft.ServiceBus"
              },
              {
                "service": "Microsoft.Sql"
              }
            ]
          },
          "dependsOn": [
            "[resourceId('Microsoft.Network/virtualNetworks', parameters('hubVnetName'))]",
            "[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('hubVnetName'), variables('mgmtSubnetName'))]",
            "[resourceId('Microsoft.Network/networkSecurityGroups', variables('nsgName'))]"
          ]
        },
        {
          "type": "subnets",
          "name": "[variables('dmzSubnetName')]",
          "apiVersion": "2019-11-01",
          "properties": {
            "addressPrefix": "[parameters('dmzSubnetPrefix')]",
            "networkSecurityGroup": {
              "id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('nsgName'))]"
            }
          },
          "dependsOn": [
            "[resourceId('Microsoft.Network/virtualNetworks', parameters('hubVnetName'))]",
            "[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('hubVnetName'), variables('mgmtSubnetName'))]",
            "[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('hubVnetName'), variables('sharedSubnetName'))]",
            "[resourceId('Microsoft.Network/networkSecurityGroups', variables('nsgName'))]"
          ]
        }
      ],
      "dependsOn": [
        "[resourceId('Microsoft.Network/networkSecurityGroups', variables('nsgName'))]"
      ]
    },

Expected response (i.e. "I expected no noise since the template has not been modified since the resources were deployed)
Clean diff
Current (noisy) response (either include a screenshot of the what-if output, or copy/paste the text)

Additional context
Add any other context about the problem here.

@majastrz @shenglol - FWIW, the subnets are being created as child resources, but ARM sees them as properties of the VNET resource. Not sure if we already have a case like this.
@tyconsulting - can you share the full template? Looks like not all subnets are being created the same way, but I am not sure.

Reporting a few more noise for properties:

properties.networkSecurityGroup:
properties.privateEndpointNetworkPolicies
properties.privateLinkServiceNetworkPolicies
properties.serviceEndpoints

  ~ Microsoft.Network/virtualNetworks/test-vnet-2 [2019-11-01]
    ~ properties.subnets: [
      ~ 0:

        ~ properties.networkSecurityGroup:

            id: "/subscriptions/<subscription id>/resourceGroups/test-vnet-rg-1/providers/Microsoft.Network/networkSecurityGroups/test-vnet-2_Subnet2-nsg"

          => "[if(or(variables('records')[0].object.createDefaultNsg,not(empty(variables('records')[0].object.networkSecurityGroupRules))),reference(concat('subnetsDeployment-',uniqueString(parameters('virtualNetworkName')),0)).outputs.networkSecurityGroup.value,json('null'))]"

        ~ properties.privateEndpointNetworkPolicies:    "Enabled" => "enabled"
        ~ properties.privateLinkServiceNetworkPolicies: "Enabled" => "enabled"
        ~ properties.serviceEndpoints: [
            0:

              service: "Microsoft.AzureActiveDirectory"

            1:

              service: "Microsoft.AzureCosmosDB"

            2:

              service: "Microsoft.EventHub"

            3:

              service: "Microsoft.KeyVault"

            4:

              service: "Microsoft.ServiceBus"

            5:

              service: "Microsoft.Storage"

            6:

              service: "Microsoft.Sql"

          ] => "[if(not(empty(variables('records')[0].object.serviceEndpoints)),reference(concat('subnetsDeployment-',uniqueString(parameters('virtualNetworkName')),0)).outputs.serviceEndpoints.value,json('null'))]"

      ~ 1:

        ~ properties.networkSecurityGroup:

            id: "/subscriptions/<subscription id>/resourceGroups/test-vnet-rg-1/providers/Microsoft.Network/networkSecurityGroups/test-vnet-2_Subnet1-nsg"

          => "[if(or(variables('records')[1].object.createDefaultNsg,not(empty(variables('records')[1].object.networkSecurityGroupRules))),reference(concat('subnetsDeployment-',uniqueString(parameters('virtualNetworkName')),1)).outputs.networkSecurityGroup.value,json('null'))]"

        ~ properties.privateEndpointNetworkPolicies:    "Enabled" => "enabled"
        ~ properties.privateLinkServiceNetworkPolicies: "Enabled" => "enabled"

     ~ 2:

        ~ properties.privateEndpointNetworkPolicies:    "Enabled" => "enabled"
        ~ properties.privateLinkServiceNetworkPolicies: "Enabled" => "enabled"

      ~ 3:

        ~ properties.privateEndpointNetworkPolicies:    "Disabled" => "disabled"
        ~ properties.privateLinkServiceNetworkPolicies: "Enabled" => "enabled"
        ~ properties.serviceEndpoints: [
            0:

              service: "Microsoft.Storage"

          ] => "[if(not(empty(variables('records')[3].object.serviceEndpoints)),reference(concat('subnetsDeployment-',uniqueString(parameters('virtualNetworkName')),3)).outputs.serviceEndpoints.value,json('null'))]"

      ]

I encounter also noises but with Microsoft.Network/virtualNetworks/virtualNetworkPeerings resource

This is tracked by https://github.com/Azure/arm-template-whatif/issues/78.

@shenglol @alex-frankel I believe, there's more going on here than "resources that manage other resources declared in the template" (#78)
virtualnetworks properties like "serviceAssociationLinks" can't be managed by the user at all (AFAIK) but are managed by other ARM providers entirely.
Hooking up an App Service to a VNET registers the link, but that process is done for a different resource in a different provider and likely in a different template.

Apologies, we misinterpreted this issue as partially being due to #78, but we're removing that link. There's a few things going on here, but thankfully @shenglol is addressing all of it now, so we should be able to clean up this response.

We committed a fix to handle the vnet/subnet relationship, and it's waiting on deployment [w40]. I'm going to work on reducing the other noise for subnet.

[w03]

[w03]

@shenglol is there an issue tracking the remaining noise or is it this one? I still get this on subnet redeployment

@JustinGrote thanks for reporting them! I'll fix the noise and we can keep this issue open.

[w11].

What is the status of this? Still seeing properties.virtualNetworkPeerings removal on what and now I'm not sure if that is noice or actual result.


Override the DbConnection to use inside ambient transaction to prevent requirement of DTC

Is there a way to set what DbConnection will be used by EF Core?
I have a situation where an ambient transaction is started.  Then a different processes (not EF Core) uses the database inside that transaction.  Then EF Core tries to make some database changes, but when it opens the connection, it gets an error about needing a Distributed Transaction Coordinator.  This is because there are two different connections trying to use the same transaction.
The solution is to tell EF Core to use the same connection that was used for the database changes earlier in the transaction.

For now I'm just using reflection to override the connection when a transaction starts and then setting it back to its original value when the transaction completes unless there is a better way:

var dataContextDatabase = dbContext.Database;
var databaseConnection = (SqlServerConnection)dataContextDatabase.GetInfrastructure().GetService(typeof(IRelationalConnection));
_EfCoreConnection = (LazyRef<DbConnection>)databaseConnection.GetFieldValue("_connection");
_EfCoreConnectionIsOwned = (bool)databaseConnection.GetFieldValue("_connectionOwned");
databaseConnection.SetFieldValue("_connection", new LazyRef<DbConnection>(_SharedSqlConnection));
databaseConnection.SetFieldValue("_connectionOwned", false);

Duplicate of #8494. But also see also discussion on #13041 and #13530 for ideas on workarounds.

@ajcvickers
Thanks!
The first solution requires Transient-scoped DbContext, and the second assumes both connections are inside EF Core, so I don't think either works for me.  I'll continue using reflection for now until this is supported.


[ENHANCEMENT]: ADD DOCKER AND MAKE IT LIGHT

Hey everyone,
I’ve been encountering several issues while trying to run the project locally—mainly dependency conflicts, outdated packages, and malloc errors, even after setting up everything as required. It’s been pretty tricky to get everything working smoothly.
To make contributing easier and avoid these environment issues, I think it would be beneficial to containerize the project using Docker. This would ensure that everyone works in a consistent environment without worrying about dependency conflicts or setup problems.
It would be great if someone from the DevOps side could step in to help set this up. We can also test the Docker setup for production to make sure it works perfectly.
Looking forward to your thoughts on this!
Cheers

I can dockerize this project, assign me so I can work on this issue.

i assigned it to you

make sure you star repo ⭐  and follow on github , giving hacktoberfest tag after this step


OTAR042 -Tractability: Co-ordinate outreach activities

[ ] Add documentation to gitbook

[ ] Amend FAQ section

[ ] Create youtube video

[ ] Blog post

[ ] Twitter/ facebook plans

[ ] Co-ordination with ChEMBL?

[ ] Update training materials

@deniseOme, do you want me to coordinate with Chris and interview him for a Q&A blog post?

For the upcoming 18.12 release, we'll be adding 3 new features related to tractability:

Adding the on-hover flower in the Prioritisation view

Adding the tractability facet to both the Prioritisation and Association views

Adding summary tractability data to the batch search page

For more information, please see issue #264


DeepBGC failed with Exception: Unexpected error detecting protein domains using HMMER hmmscan

Hi, I have some problems about using DeepBGC to identify the BGCs in Mycobacterium tuberculosis H37Rv (high GC Gram+).
Here is my DeepBGC version information:
(base) mima@123456:~/BGC/jieheganjun$ deepbgc info
 _____                  ____    ____   ____
 |  _ \  ___  ___ ____ | __ )  / ___) / ___)
 | | \ \/ _ \/ _ \  _ \|  _ \ | |  _ | |
 | |_/ /  __/  __/ |_) | |_) || |_| || |___
 |____/ \___|\___| ___/|____/  \____| \____)
=================|_|===== version 0.1.26 =====
INFO    08/04 16:55:44   Available data files: ['Pfam-A.31.0.hmm', 'Pfam-A.31.0.hmm.h3f', 'Pfam-A.31.0.hmm.h3m', 'Pfam-A.31.0.hmm.h3i', 'Pfam-A.31.0.clans.tsv', 'Pfam-A.31.0.hmm.h3p']
INFO    08/04 16:55:44   ================================================================================
INFO    08/04 16:55:44   Available detectors: ['clusterfinder_retrained', 'clusterfinder_original', 'deepbgc', 'product_class', 'product_activity', 'clusterfinder_geneborder']
INFO    08/04 16:55:44   --------------------------------------------------------------------------------
INFO    08/04 16:55:44   Model: clusterfinder_retrained
INFO    08/04 16:55:44   Loading model from: /home/mima/.local/share/deepbgc/data/0.1.0/detector/clusterfinder_retrained.pkl
WARNING 08/04 16:55:44   Model not supported: ('Package "hmmlearn" needs to be installed to run ClusterFinder HMM. ', 'Install extra dependencies using: \n    pip install "deepbgc[hmm]"')
INFO    08/04 16:55:44   --------------------------------------------------------------------------------
INFO    08/04 16:55:44   Model: clusterfinder_original
INFO    08/04 16:55:44   Loading model from: /home/mima/.local/share/deepbgc/data/0.1.0/detector/clusterfinder_original.pkl
WARNING 08/04 16:55:44   Model not supported: ('Package "hmmlearn" needs to be installed to run ClusterFinder HMM. ', 'Install extra dependencies using: \n    pip install "deepbgc[hmm]"')
INFO    08/04 16:55:44   --------------------------------------------------------------------------------
INFO    08/04 16:55:44   Model: deepbgc
INFO    08/04 16:55:44   Loading model from: /home/mima/.local/share/deepbgc/data/0.1.0/detector/deepbgc.pkl
Using TensorFlow backend.
WARNING 08/04 16:55:49   From /home/mima/miniconda3/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING 08/04 16:55:49   From /home/mima/miniconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
INFO    08/04 16:55:50   Type: KerasRNN
INFO    08/04 16:55:50   Version: 0.1.0
INFO    08/04 16:55:50   Timestamp:<PHONE_NUMBER>.986168 (2019-02-28T06:14:27.986168)
INFO    08/04 16:55:50   --------------------------------------------------------------------------------
INFO    08/04 16:55:50   Model: product_class
INFO    08/04 16:55:50   Loading model from: /home/mima/.local/share/deepbgc/data/0.1.0/detector/product_class.pkl
/home/mima/miniconda3/lib/python3.7/site-packages/sklearn/base.py:306: UserWarning: Trying to unpickle estimator DecisionTreeClassifier from version 0.18.2 when using version 0.21.3. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)
/home/mima/miniconda3/lib/python3.7/site-packages/sklearn/base.py:306: UserWarning: Trying to unpickle estimator RandomForestClassifier from version 0.18.2 when using version 0.21.3. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)
INFO    08/04 16:55:50   Type: RandomForestClassifier
INFO    08/04 16:55:50   Version: 0.1.0
INFO    08/04 16:55:50   Timestamp:<PHONE_NUMBER>.019103 (2019-03-05T18:23:30.019103)
INFO    08/04 16:55:50   --------------------------------------------------------------------------------
INFO    08/04 16:55:50   Model: product_activity
INFO    08/04 16:55:50   Loading model from: /home/mima/.local/share/deepbgc/data/0.1.0/detector/product_activity.pkl
INFO    08/04 16:55:50   Type: RandomForestClassifier
INFO    08/04 16:55:50   Version: 0.1.0
INFO    08/04 16:55:50   Timestamp:<PHONE_NUMBER>.886473 (2019-03-05T18:23:53.886473)
INFO    08/04 16:55:50   --------------------------------------------------------------------------------
INFO    08/04 16:55:50   Model: clusterfinder_geneborder
INFO    08/04 16:55:50   Loading model from: /home/mima/.local/share/deepbgc/data/0.1.0/detector/clusterfinder_geneborder.pkl
WARNING 08/04 16:55:50   Model not supported: ('Package "hmmlearn" needs to be installed to run ClusterFinder HMM. ', 'Install extra dependencies using: \n    pip install "deepbgc[hmm]"')
INFO    08/04 16:55:50   ================================================================================
INFO    08/04 16:55:50   Available classifiers: ['product_class', 'product_activity']
INFO    08/04 16:55:50   --------------------------------------------------------------------------------
INFO    08/04 16:55:50   Model: product_class
INFO    08/04 16:55:50   Loading model from: /home/mima/.local/share/deepbgc/data/0.1.0/classifier/product_class.pkl
INFO    08/04 16:55:50   Type: RandomForestClassifier
INFO    08/04 16:55:50   Version: 0.1.0
INFO    08/04 16:55:50   Timestamp:<PHONE_NUMBER>.019103 (2019-03-05T18:23:30.019103)
INFO    08/04 16:55:51   --------------------------------------------------------------------------------
INFO    08/04 16:55:51   Model: product_activity
INFO    08/04 16:55:51   Loading model from: /home/mima/.local/share/deepbgc/data/0.1.0/classifier/product_activity.pkl
INFO    08/04 16:55:51   Type: RandomForestClassifier
INFO    08/04 16:55:51   Version: 0.1.0
INFO    08/04 16:55:51   Timestamp:<PHONE_NUMBER>.886473 (2019-03-05T18:23:53.886473)
INFO    08/04 16:55:51   ================================================================================
WARNING 08/04 16:55:51   Some warnings detected, check the output above

Then type the command deepbgc pipeline ./GCF_000195955.2_ASM19595v2_genomic.fna. Unfortunately, it failed.
(base) mima@123456:~/BGC/jieheganjun$ deepbgc pipeline ./GCF_000195955.2_ASM19595v2_genomic.fna
 _____                  ____    ____   ____
 |  _ \  ___  ___ ____ | __ )  / ___) / ___)
 | | \ \/ _ \/ _ \  _ \|  _ \ | |  _ | |
 | |_/ /  __/  __/ |_) | |_) || |_| || |___
 |____/ \___|\___| ___/|____/  \____| \____)
=================|_|===== version 0.1.26 =====
INFO    08/04 16:32:36   Loading model from: /home/mima/.local/share/deepbgc/data/0.1.0/detector/deepbgc.pkl
Using TensorFlow backend.
WARNING 08/04 16:32:36   From /home/mima/miniconda3/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING 08/04 16:32:36   From /home/mima/miniconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
INFO    08/04 16:32:37   Loading model from: /home/mima/.local/share/deepbgc/data/0.1.0/classifier/product_class.pkl
/home/mima/miniconda3/lib/python3.7/site-packages/sklearn/base.py:306: UserWarning: Trying to unpickle estimator DecisionTreeClassifier from version 0.18.2 when using version 0.21.3. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)
/home/mima/miniconda3/lib/python3.7/site-packages/sklearn/base.py:306: UserWarning: Trying to unpickle estimator RandomForestClassifier from version 0.18.2 when using version 0.21.3. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)
INFO    08/04 16:32:37   Loading model from: /home/mima/.local/share/deepbgc/data/0.1.0/classifier/product_activity.pkl
INFO    08/04 16:32:37   Processing input file 1/1: ./GCF_000195955.2_ASM19595v2_genomic.fna
INFO    08/04 16:32:37   ================================================================================
INFO    08/04 16:32:37   Processing record #1: NC_000962.3
WARNING 08/04 16:32:37   Updating record alphabet to generic_dna
INFO    08/04 16:32:37   Finding genes in record: NC_000962.3
INFO    08/04 16:32:47   Detecting Pfam domains in "NC_000962.3" using HMMER hmmscan, this might take a while...
WARNING 08/04 16:42:14   == HMMER hmmscan Error: ================
WARNING 08/04 16:42:14
WARNING 08/04 16:42:14   == End HMMER hmmscan Error. ============
ERROR   08/04 16:42:14   Unexpected error detecting protein domains using HMMER hmmscan
Traceback (most recent call last):
  File "/home/mima/miniconda3/lib/python3.7/site-packages/deepbgc/main.py", line 113, in main
    run(argv)
  File "/home/mima/miniconda3/lib/python3.7/site-packages/deepbgc/main.py", line 102, in run
    args.func.run(**args_dict)
  File "/home/mima/miniconda3/lib/python3.7/site-packages/deepbgc/command/pipeline.py", line 177, in run
    step.run(record)
  File "/home/mima/miniconda3/lib/python3.7/site-packages/deepbgc/pipeline/annotator.py", line 35, in run
    pfam_annotator.annotate()
  File "/home/mima/miniconda3/lib/python3.7/site-packages/deepbgc/pipeline/pfam.py", line 97, in annotate
    self._run_hmmscan(protein_path, domtbl_path)
  File "/home/mima/miniconda3/lib/python3.7/site-packages/deepbgc/pipeline/pfam.py", line 73, in _run_hmmscan
    raise Exception("Unexpected error detecting protein domains using HMMER hmmscan")
Exception: Unexpected error detecting protein domains using HMMER hmmscan
ERROR   08/04 16:42:14   ================================================================================
ERROR   08/04 16:42:14   DeepBGC failed with Exception: Unexpected error detecting protein domains using HMMER hmmscan
ERROR   08/04 16:42:14   ================================================================================

I don't know how to solve this problem, so please help me. (By the way, I have tried put this genome sequences into antiSMASH, and it works). Thanks in advance.

Hi @Bollie15, I tried running deepbgc on those sequences and I didn't get any error. Can you try running again? Looks like hmmscan has crashed, maybe not enough memory was available?

Hi @Bollie15, I tried running deepbgc on those sequences and I didn't get any error. Can you try running again? Looks like hmmscan has crashed, maybe enough memory was not available?

Thanks! It ran successfully after I closed all background programs.


⚠ Bofrost.gr has degraded performance

In b69bcb3, Bofrost.gr (https://www.bofrost.gr) experienced degraded performance:

HTTP code: 200
Response time: 3184 ms

Resolved: Bofrost.gr performance has improved in bb29b48 after 7 minutes.


refactor sq api init

Makes initializing the SquirrelManagers more readable by having definitions for CLIENT, UI & SERVER vm in one place.
Currently it looks like this
ON_DLL_LOAD_RELIESON("client.dll", ClientSquirrel, ConCommand, (CModule module))
{
	// ...

	g_pSquirrel<ScriptContext::CLIENT>->__sq_defconst = module.Offset(0x12120).As<sq_defconstType>();
	g_pSquirrel<ScriptContext::UI>->__sq_defconst = g_pSquirrel<ScriptContext::CLIENT>->__sq_defconst;

	// repeat for every function ...
}

ON_DLL_LOAD_RELIESON("server.dll", ClientSquirrel, ConCommand, (CModule module))
{
	g_pSquirrel<ScriptContext::SERVER>->__sq_defconst = module.Offset(0x12120).As<sq_defconstType>();

	// repeat for every function ...
}

The redundancy is annoying and makes it hard to see at a glance what the offsets for the functions are.
This PR implements this function that initializes the functions either on the CLIENT & UI or on the SERVER dynamically:
// Initialize a function on the CLIENT and UI or SERVER, depending on context
template <ScriptContext context, typename T>
void SetSharedMember(SquirrelManager<context>* manager, T* member, const CModule module, const int clOffset, const int svOffset)
{
	switch (context)
	{
	case ScriptContext::CLIENT:
	{
		*member = module.Offset(clOffset).As<T>(); // Set the member for the CLIENT vm
		*(T*)((char*)g_pSquirrel<ScriptContext::UI> + (int64_t)member - (int64_t)manager) = *member; // Set the member for the UI vm
		break;
	}
	case ScriptContext::SERVER:
		*member = module.Offset(svOffset).As<T>();
	}
}

Initializing a function for all vms now looks like this:
template <ScriptContext context> void InitSharedSquirrelFunctions(SquirrelManager<context>* manager, CModule module)
{
	SetSharedMember<context, sq_defconstType>(manager, &manager->__sq_defconst, module, 0x12120, 0x1F550);
}

The issue with this is that the memory layout of classes is ub afaik

*bump*

@uniboi just a heads-up that this PR has some merge conflicts atm

merge conflicts still


improve test coverage

Codecov Report

Merging #9 (5e75153) into main (e6cd826) will increase coverage by 5.29%.
The diff coverage is 50.00%.

@@            Coverage Diff             @@
##             main       #9      +/-   ##
==========================================
+ Coverage   60.00%   65.29%   +5.29%
==========================================
  Files          11       11
  Lines         340      340
==========================================
+ Hits          204      222      +18
+ Misses        107       88      -19
- Partials       29       30       +1

Impacted Files
Coverage Δ

cmd/squealer/main.go
29.82% <50.00%> (+29.82%)
:arrow_up:

internal/app/squealer/mertics/metrics.go
81.81% <0.00%> (+9.09%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e6cd826...5e75153. Read the comment docs.


Issue-5165 Manifest merge tool can't merge arrays in plist

More info about the issue is available here https://github.com/defold/defold/issues/5165
This issue is similar to the issue we had in bob earlier: https://github.com/defold/defold/issues/4977
Current result:

fix https://github.com/defold/defold/projects/40


[GLUTEN-4660][CH]asynchronous SourceFromJavaIter

What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
Fixes: #4660
How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
unit tests
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

before

after


Fix Subaccount Cleanup Job

Description
Changes proposed in this pull request:

refactor code: replace recursion with loop,
handle rate limiting,
add unit tests.

Related issue(s)

/retest

/retest


Updates botocore, certifi, chardet, docutils, idna and psutil

Adds urllib3
Removes future

Codecov Report

Merging #267 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #267   +/-   ##
=======================================
  Coverage   92.73%   92.73%
=======================================
  Files          42       42
  Lines        2753     2753
=======================================
  Hits         2553     2553
  Misses        200      200

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 14589bd...91b5d3b. Read the comment docs.


add: K3D_FIX_MOUNTS fix to make / rshared (e.g. to make Cilium work)

What
New env var K3D_FIX_MOUNTS, set it to e.g. K3D_FIX_MOUNTS=1 to make / in the node containers rshared mounts.
This fixes e.g. the setup of Cilium.
If there are no bad implications of this and no one raises issues while using it throughout a few versions, we can add it as a default entrypoint.
Why
#1063
Implications

Wow waiting so long for this to get cilium working out of the box working

@iwilltry42 , how can I set this env var upon creation of the cluster?
or is this not necessary, since #2246 was implemented?
Sorry, new to k3d ... and a bit confused

@deyanp just export it in your shell before running k3d 👍
E.g.export K3D_FIX_MOUNTS=1.
It will be enabled by default in the next release.

Wow, thanks @iwilltry42 !
I was thinking I should do
k3d cluster create $clusterName --registry-use $registryName:$registryPort --k3s-arg "--disable=traefik@server:*" --env 'K3D_FIX_MOUNTS=1@server:*'


Migrate from jCenter to mavenCentral

Need migration of splitties packages, not available in repositories of mavenCentral.
JCenter deprecation and end of service: https://developer.android.com/studio/build/jcenter-migration
On Sync gradle receives:
Failed to resolve: com.journeyapps:zxing-android-embedded:3.4.0

See #606.
Version 3.4.0 has not been migrated - can you upgrade to a later version?

@rkistner Good, thanks!


CycloneDX Presenter

Support output of CycloneDX compliant BoM. This can and probably should include the BOM Descriptor extension.
Core CycloneDX:
https://cyclonedx.org
Specification: https://github.com/CycloneDX/specification
BOM Descriptor Extension: https://cyclonedx.org/ext/bom-descriptor
Dependency Graph Extension: https://cyclonedx.org/ext/dependency-graph

This is fantastic. Once the next release is available that includes CycloneDX support, I'll add Syft to https://cyclonedx.org/tool-center/


macOS codesigning - locally messes up Keychain?

I ran the dist build ...  with codesigning locally, and it appears to have messed up my keychain considerably :)
Maybe it could either be resetted, or defaults not changed?

I think something like:
# First reset to login keychain as default
security default-keychain -s ~/Library/Keychains/login.keychain-db

Will restore an OK state. Although locally my login.keychain-db was also renamed to login_renamed.keychain-db which could have been somethign that I did at some point, so not sure if that problem originated here.


configuring resource limits in values.yaml are only applied on 1 of 2 containers

values.yaml:

resources:
    memory: 50Mi
  requests:
    cpu: 50m
    memory: 50Mi
cloudsqlproxy:
  enabled: true

kubectl.exe get deployment

template:
  spec:
    containers:
    - name: prometheus-mysql-exporter
      resources: {}
    - name: cloudsql-proxy
      resources:
        limits:
          memory: 50Mi
        requests:
          cpu: 50m
          memory: 50Mi

I would expect the resource limits applied to both prometheus-mysql-exporter and cloudsql-proxy

@stevenderycke you should follow the guidelines when opening issues if you would like the maintainers of the specific chart to be able to find this issue and reply.
I recommend also @-ing one of the maintainers found in the OWNERS fike in the chart directory


[Async] Add cached stream

We need to introduce a cache for the case when iterating over a similar stream multiple times is expensive


Type declaration file is causing errors in Angular 5

The type declaration file included in the cytoscape-node-html-label package.json angers Angular 5.
ERROR in node_modules/@types/node/index.d.ts(87,13): error TS2451: Cannot redeclare block-scoped variable 'require'.
node_modules/@types/node/index.d.ts(99,13): error TS2451: Cannot redeclare block-scoped variable 'module'.
node_modules/cytoscape-node-html-label/dist/cytoscape-node-html-label.d.ts(1,15): error TS2451: Cannot redeclare block-scoped variable 'require'.
node_modules/cytoscape-node-html-label/dist/cytoscape-node-html-label.d.ts(2,15): error TS2451: Cannot redeclare block-scoped variable 'module'.
node_modules/cytoscape-node-html-label/dist/cytoscape-node-html-label.d.ts(4,15): error TS2451: Cannot redeclare block-scoped variable 'cytoscape'.
node_modules/cytoscape-node-html-label/dist/cytoscape-node-html-label.d.ts(16,19): error TS2451: Cannot redeclare block-scoped variable 'cytoscape'.
When I remove the "typings": "./dist/cytoscape-node-html-label.d.ts" from the cytoscape-node-html-label package.json everything works.  Would it be possible, at the very least, to give us the options of installing the type declaration file separately instead of including it in the package?
Thanks,
-Chris

Now when I import nodeHtmlLabel from 'cytoscape-node-html-label';
I get: "File '.../node_modules/cytoscape-node-html-label/dist/cytoscape-node-html-label.d.ts' is not a module."

Now when I import nodeHtmlLabel from 'cytoscape-node-html-label';
I get: "File '.../node_modules/cytoscape-node-html-label/dist/cytoscape-node-html-label.d.ts' is not a module."

Got a temp fix by creating a postinstall script at package.json
"postinstall": "rm -rf ./node_modules/cytoscape-node-html-label/dist/cytoscape-node-html-label.d.ts"


 Check if the specified host path exists and is the expected type

You have to create the files/directories first. In this case, in /root/pipu/homepage you create config.json, icons, etc.


Store: error parsing block

Thanos version used
thanos, version 0.7.0 (branch: HEAD, revision: c6eaf68bec32aefd280318eaef6121d6ddb20d10)   build user:       root@aa355daaf987   build date:       20190902-15:31:30   go version:       go1.12.5

What happened
Thanos store couldn't present all the data we were expecting as we have months of data and only one day was being shown. After looking at the logs all our thanos store instances were showing an error when trying to read the meta.json file from some folders in the data directory. After inspecting the folders we confirmed that they were empty.
After the clean up of the empty folders store returned to its normal behaviour.
What you expected to happen
The folders should not be empty so some process must have failed and the folders were not cleaned up. Also, thanos probably should keep reading the folders until a new healthy one is found so the query would be correctly precessed.
How to reproduce it (as minimally and precisely as possible):
Couldn't reproduce the problem
Full logs to relevant components
level=warn ts=2019-09-20T09:53:44.795446173Z caller=bucket.go:325 msg="error parsing block range" block=01DN3WMQJ1EQM192SSTXSBPS4V err="read meta.json: open /mnt/prometheus/store/01DN3WMQJ1EQM192SSTXSBPS4V/meta.json: no such file or directory"

Anything else we need to know
We use azure blob storage with thanos.
We use store in our prd environment with thanos 0.6.0 and the problem hasn't happened.
thanos: uname -a
Linux thanos 3.10.0-957.12.1.el7.x86_64 #1 SMP Mon Apr 29 14:59:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Hello, seems like it was fixed by this: https://github.com/thanos-io/thanos/pull/1505. Please try out the newest (master) version! Shout loudly at me if it happens again with that commit.

Tested with the newest version and it seems to be solved! Thanks 😄

Yes, we are running it in production and the issue does not happen anymore.
Thanks for verifying!


Upgrade react dependencies to 15.0.1

Hello,
I want to use your library in my project, but I use react 15.0.1 and can't downgrade to 15.0.0 because of bug in this version. It's possible to update yours dependencies to 15.0.1 ?
Thanks

This shouldn't be an issue. Are you getting an error or something?

@DmitroPustovit: react is just a peer dependency, meaning that react-virtualized doesn't install it; it just expects you to. As for the version, my package.json is setup to match with  "react": "^0.14.0 || ^15.0.0" which means any bug-fix version of 0.14.x as well as 15.0.x and up. You should be able to use 15.0.1 just fine but you'll need to install it yourself, with your own package.json file.


Cookie consent image causes small CLS

@paolomoz - with regards to the image for the Cookie Preference icon, its an issue with the image itself. Despite decreasing the size of the DIV, still that image will pop up like that on the right and the bottom since that image cuttings are like that only. Please check the attached screenshots -

I think this is fixed with https://github.com/hlxsites/ororafresh/pull/117


Is v1.0.0 stable and usable enough for basic function of watching a file?

This library has what I'm looking boost.asio approach and xplat support , but sadly no recent activity.  Seems like it had good momentum but the PR's are 18 months old, which is quite a bummer.  Please advise on 1.0's stability and the likelihood of revival.

Hi, I'm planning to return to it soon, but it has been and is enough for watching file changes - been using it myself for a while.

I'll close this one.


hotkeys: Add hotkeys for navigating Streams menu.

Adds hotkeys for left/right arrow keys to switch tabs in Streams menu (#4228)
Adds hotkey U to subscribe/unsubscribe from selected stream in Streams menu (#4229)
Moves cursor to "Filter streams" input if current selected stream is the first in the Streams menu (#4227)
Adds hotkey V for viewing/narrowing to messages of selected stream in Streams menu

@synicalsyntax one bug that I'm seeing is the following.
In the "Subscribed" tab, pick a stream (say devel stream) and use the hotkey U to unsubscribe from it.
Now try moving up and down past where that stream used to be. You can't move past where the devel stream used to be.
Similarly, if you unsubscribed from the first stream and try to move up and down, it doesn't work.

@lonerz Fixed! :)

@timabbott Fixed :)

@lonerz Fixed, although I didn't get any errors in my previous build (it passed successfully); maybe it's an error in the linter?

@synicalsyntax possibly, not sure.

Nice, merged, thanks @synicalsyntax!  This feels a lot nicer :).
I think there's a few follow-up things in this space that we should do though:

When you open the streams page the first time, the "filter streams" text box should already have focus.
Is "U" the best key for "subscribe/unsubscribe"?  I'm now wondering whether we should do "S" instead.
Let's open an issue for the fact that unsubscribing with hotkeys while in the 'Subscribed' tab needs to disappear the row; I don't think that's optimal and probably reflects some sort of subtle bug in how that page works.
Right now, we're fetching subscribers from the server on every navigation, which could be really expensive.  I think we actually have all the data on the frontend to render subscribers without contacting the server.  I opened https://github.com/zulip/zulip/issues/4314 for this.
We need to document these!  I think we want to make a new "Streams overlay" section in the docs.

Leaving this PR open so we can just handle some of the quick things by your just pushing more to this PR :).

@timabbott Some clarifications on two of the improvements you requested:

the S shortcut is already used by another action, I don't think we've fully added support for hotkeys with multiple actions but I could try implementing it.
I enforced the disappearing rows in the unsubscribing hotkey. Weird behaviors happen if you don't remove it once you unsubscribe:

As you can see, the row suddenly disappeared when I tried subscribing to it again. Therefore I made sure that the row would disappear upon its unsubscription if the Subscribed tab was open.

is that what you what you meant by "When you open the streams page the first time, the "filter streams" text box should already have focus"?

Yep, that's what I meant.  I tried this out and there's one problem -- if you use the left sidebar popovers to go to an individual stream's settings page, the "filter streams" text box probably shouldn't be focused (since you have a specific stream selected), but it is in this case...

Weird behaviors happen if you don't remove it once you unsubscribe

Right, what I'm saying is that I think we should at least open an issue for fixing that weird behavior bug; it happens with clicking too.  Once we've fixed it we can make the hotkey not disappear the row (I agree given that bug your implementation for the hotkey is the best we can do).

@timabbott Sorry for the long delay in fixing this, but the Filter streams input should now only occur when the Streams modal is launched without a stream row selected. :)
I'll like to wait until #4778 gets merged to prevent merge conflicts while trying to document these keyboard shortcuts, also.

@synicalsyntax I think S should stay as message narrow, since s is stream narrow, which makes sense (s for stream). T seems off (I would say the other keys would be n or m).

For the second commit, it seems like we probably don't need to change the existing use of S; we just need to make that key do 2 things.
I guess the main downside is that our current naming structure kinda assumes that we're not going to have a single key do different things in different contexts; it might make sense to plan to restructure that model.

@timabbott Changed the hotkey to S, after a long struggle with getting tests to pass :)
I also refactored the Keyboard shortcuts documentation to be less wordy like you asked me too.

@synicalsyntax this looks great!  I merged it.  A couple quick issues I noticed we should potentially fix in a follow-up:

It seems like the S hotkey only works to unsubscribe; it doesn't work in the "all streams" tab?
The centered 'Streams' hotkeys looks a bit funny without something next to it, which we need to figure out how to solve.


[null-ls] failed to run generator: ...go/null_ls.lua:201: attempt to index field 'golangci_lint' (a nil value)

Thanks a lot for making this plugin!
Recently I have encountered the above error when using this config derived from the project README's null-ls section:
  {
    "nvimtools/none-ls.nvim",
    opts = function(_, opts)
      if not opts.sources then opts.sources = {} end
      local go_nls = require("go.null_ls")
      opts.sources = vim.list_extend(opts.sources, {
        go_nls.golangci_lint(),
      })
      return opts
    end,
  },

Looking at the golangci_lint() function being called:
https://github.com/ray-x/go.nvim/blob/e80846dec565dac6418334cfc4eecc087678ff69/lua/go/null_ls.lua#L174-L175
... and particularly L201:
https://github.com/ray-x/go.nvim/blob/e80846dec565dac6418334cfc4eecc087678ff69/lua/go/null_ls.lua#L201-L207
... it looks like _GO_NVIM_CFG.null_ls exists, but not _GO_NVIM_CFG.null_ls.golangci_lint (so this will generate a nil), and _GO_NVIM_CFG.null_ls.golangci_lint.disable will cause an index error (same for L204). The current code has failed to handle this case.
:lua = _GO_NVIM_CFG
{
  build_tags = "",
  comment_placeholder = "   ",
  dap_debug = true,
  dap_debug_gui = {},
  dap_debug_keymap = true,
  dap_debug_vt = {},
  dap_port = 38697,
  dap_retries = 20,
  dap_timeout = 15,
  diagnostic = {},
  disable_defaults = false,
  disable_per_project_cfg = false,
  fillstruct = "gopls",
  floaterm = {},
  go = "go",
  go_boilplater_url = "https://github.com/thockin/go-build-template.git",
  go_input = <function 1>,
  go_select = <function 2>,
  gocoverage_sign = "█",
  gocoverage_skip_covered = false,
  gofmt = "gopls",
  goimports = "gopls",
  gopls_remote_auto = true,
  gotests_template = "",
  gotests_template_dir = "",
  icons = {},
  iferr_vertical_shift = 4,
  log_path = "/Users/rami3l/tmp/gonvim.log",
  lsp_cfg = false,
  lsp_codelens = true,
  lsp_diag_update_in_insert = false,
  lsp_document_formatting = true,
  lsp_fmt_async = false,
  lsp_gofumpt = false,
  lsp_inlay_hints = {},
  lsp_keymaps = true,
  luasnip = false,
  max_line_len = 0,
  null_ls = {},
  null_ls_document_formatting_disable = false,
  on_exit = <function 3>,
  on_jobstart = <function 4>,
  on_stderr = <function 5>,
  on_stdout = <function 6>,
  run_in_floaterm = false,
  sign_covered_hl = "String",
  sign_partial_hl = "WarningMsg",
  sign_priority = 7,
  sign_uncovered_hl = "Error",
  tag_options = "json=omitempty",
  tag_transform = false,
  test_efm = false,
  test_runner = "go",
  textobjects = true,
  trouble = false,
  useremail = "",
  username = "",
  verbose = false,
  verbose_tests = false
}

FWIW I'm using AstroNvim's Golang pack as my base config on Neovim v0.10, if this helps.

I think   null_ls = {},       reset the settings
the default is
 null_ls = {
    golangci_lint = {
      -- disable = {'errcheck', 'staticcheck'}, -- linters to disable empty by default
      -- enable = {'govet', 'ineffassign','revive', 'gosimple'}, -- linters to enable; empty by default
    },
  },

I think null_ls = {}, reset the settings the default is
 null_ls = {
    golangci_lint = {
      -- disable = {'errcheck', 'staticcheck'}, -- linters to disable empty by default
      -- enable = {'govet', 'ineffassign','revive', 'gosimple'}, -- linters to enable; empty by default
    },
  },

@ray-x Thanks for the prompt reply! Looks like this has something to do with disable_defaults = true in the base config. Is this breakage intentional?
https://github.com/ray-x/go.nvim/blob/e80846dec565dac6418334cfc4eecc087678ff69/lua/go.lua#L201-L210

It looks like 294d65c93514f14fbbe8af0545ab8918d939acdb has addressed the issue for me. Many thanks :)

@ray-x just a heads up, this breaks the diagnostics disabling when calling disable_defaults. This is because diagnostics should be getting set to false.

I'm not sure exactly if it breaks other default disabling, but it seems like not a very robust approach when the values that are expected for disabling differ between the options

It is a bug when a value can be either bool or list. I pushed a fix for it. 4274cb3fb31113c93485877d0d2b77cc80383da9


Simulated Address Info in Live Order

After successfully completing a live order in our staging environment, we observed an incorrect Shipping Address in the confirmation email that was sent from PayPal. The address was incorrectly using info from simulated_address for Name and Street Address, instead of using the real information that we had entered as part of our order.
https://monosnap.com/file/6f8t6lmOwjoPMKF2EekmstRUsX99FN

Hey Thomas, thanks for this report! PayPal frustratingly returns a partial address when the user logs in and expects us to recalculate shipping off of that, so that's where the "fake street" comes in - it definitely shouldn't be getting returned back to PayPal and stored as the users actual address though. Looking into it now!

Thanks a lot @seand7565 -- Looking forward to trying this. Might just wait for the merge if you expect it soon.

@MFRWDesign It's been merged! That should fix this issue, though I'll let you try it out and double check before closing this issue. 😄

@MFRWDesign I can't remember if you told me this issue was fixed or not, but I'll close it for now - let me know if you still experience this issue. Thanks!


Add Esuhai Technology

@tinleesutech ESUHAI TECHNOLOGY seems to be not a school. I have to close this request without merging.
Shall I be mistaken, please provide us with:

the school official website URL
the school street address, including city and country
an URL of a page on the official website where the school offers an IT-related long-term (>1 year) course
a proof, which shows that the school recognizes the domain you are submitting as an official email domain for the students.

Please provide us with a proof, which shows that the school recognizes the domain you are submitting as an official email domain for the students.
You can provide us with any of the following:

A copy of instruction (a hard-copy or electronic one) given to you by the school that tells you to use an email address in this email domain
An URL of a page at the official school website that says that students use this email domain
A reference to a recognized journal where an article was published by someone from the school, with explicit affiliation to the school, and e-mail address from the email domain
A WHOIS data that link the email domain to the school somehow
A copy of official letter from the school representative saying that the school provides e-mail addresses in this domain to the students
Anything else that can prove this fact in similar way, i.e. can be verified independently.


Fix follower stats SQL query

The error "org.postgresql.util.PSQLException: ERROR: subquery in FROM must have an alias\n Hint: For example, FROM (SELECT ...) [AS] foo.\n Position: 123" can likely be fixed by changing line 990 in WebApplication.java CAST(followings/? AS INT) to FLOOR(followings/? AS INT) as PostgreSQL don't need to use CAST like SQLite does for this.

#175


[#127073275] Replace tests that use Mimic with Ginkgo/Gomega

What
Create new Ginkgo suite for concourse/scripts
This is the first step at replacing some (and later all) of our tests for
scripts under concourse/scripts/ to use Ginkgo and Gomega instead of Rspec
and Aruba+Mimic. We're not terribly happy with the current libraries
because:
Aruba magically hooks into Rspec and is quite complex. The way we're using
it through Rspec (instead of Cucumber, which I'd rather avoid) doesn't seem
to be that commonly used, which is reflected in the unclear API and
documentation. Several people in the team have struggled to write new tests
for scripts because of these issues.
Mimic seems the best choice for creating a real HTTP server with stub
responses that is accessed by a different process in Ruby. However we've run
into problems with it not listening on the correct interface, not being able
to stub 404 bodies, not being able to assert on the contents of requests,
and not being able to query the address it's bound to. We have raised PRs
for some of these issues, but it's unclear whether the project is
maintained.
Alex pointed out that Ginkgo and Gomega have some very good libraries for
executing binaries (gexec) and creating HTTP servers (ghttp). Using them
feels far less "magic" than Aruba and Mimic. I think it's a good fit for us
because we also have other tests in Ginkgo, so we're familiar with the
tooling, and using a separate language from the scripts (which are mostly
shell and Ruby) will help enforce blackbox testing (which is most important
for these scripts where we care about the input/output/exit code).
The only oddity is that we'll need to call the scripts with bundle exec
because we're no longer running them from within a Bundler session.
I'm focussing on converting the scripts that use Mimic at the moment because
the listen bug is preventing us from running them locally under OS X. If
this approach proves successful then we should write all new tests that
would have used Aruba in Ginkgo instead and slowly transition existing tests
over. I have no intention of changing other Rspec tests which don't use
Aruba. So the focus is primarily on concourse/scripts/.
Travis only lets you specify a single language for a build. With most of our
other unit tests still being written in Ruby we should keep that as our
primary language for Travis. Go is available in that container, but it's an
older version of 1.4 which might not be supported by Ginkgo in future (they
dropped 1.3). So I'm installing a later version using the gimme tool which
Travis use in Go builds. We also need to install Ginkgo and other
dependencies.
Convert bosh_pre_destroy tests to Ginkgo
For the reasons described in the preceding commit which adds a test suite
for the concourse/scripts/ directory.
The structure and assertions have been copied pretty much like-for-like
which will make it easier to compare whether we are testing the same things.
Convert kibana_set_utc tests to Ginkgo
For the reasons described in the preceding commit which adds a test suite
for the concourse/scripts/ directory.
The structure and assertions have been copied pretty much like-for-like
which will make it easier to compare whether we are testing the same things.
Though please note the following:

I've split the "if " and "it " descriptions out to separate
Context and It blocks so that I have somewhere to put the
BeforeEach.
Context blocks could be DRYed at a later date.
The assertions for ReceivedRequests() aren't as valuable now that we're
asserting the contents of requests and ghttp will assert that undefined
stubs aren't hit.
I've added assertions for STDOUT and STDERR content which we didn't have
before and I believe is important to tell how the script is working.

How to review

Compare the new tests with the previous tests and confirm that they cover the same functionality.
Confirm that the Travis build passes.
Run the tests locally using make test or make spec.

I'm also interested in feedback from more than one person about:

Are the tests any harder or easier to understand?
Are the tests any harder to run locally?

Who can review
Not @dcarley

This approach looks good to me. I've run into a couple of issues with the implementation though:

both tests fail with errors like bundler: command not found: bosh_pre_destroy.rb. This can be fixed by replacing the command with ./bosh_pre_destroy.rb. This feels more correct to me anyway because the scripts directory won't be on the PATH. It seems that newer versions of bundler may have started implicitly looking in the current directory.

I've intermittently had the following error:

BoshPreDestroy
/Users/alex/paas/paas-cf/concourse/scripts/bosh_pre_destroy_test.go:94
no deployments
/Users/alex/paas/paas-cf/concourse/scripts/bosh_pre_destroy_test.go:74
should return a zero exit code and no output [It]
/Users/alex/paas/paas-cf/concourse/scripts/bosh_pre_destroy_test.go:73
Timed out after 1.001s.
Expected process to exit.  It did not.

/Users/alex/paas/paas-cf/concourse/scripts/bosh_pre_destroy_test.go:70

I reckon the default timeout (1s) is too short here.

* I get the following error from the `kibana_set_utc` test.

KibanaSetUtc
/Users/alex/paas/paas-cf/concourse/scripts/kibana_set_utc_test.go:210
elastic search is not available
/Users/alex/paas/paas-cf/concourse/scripts/kibana_set_utc_test.go:159
reports an error [It]
/Users/alex/paas/paas-cf/concourse/scripts/kibana_set_utc_test.go:158
Got stuck at:

Waiting for:
    bundler: failed to load command

Again, I reckon this is a string that's only output with newer versions of bundler.

For reference, I'm running bundler 1.11.2.

Looking into the problem described above..

I've rebased with the following changes:

added a path to all execs
kibana_set_utc.rb is no longer run through bundle exec. I think you're right, that there's no need and very little danger if we're not using third-party gems. It speeds up the execution a bit and no longer pollutes STDOUT/STDERR. I've changed the STDOUT/STDERR assertions to account for this. My only concern is how will other people adding new tests know which example to copy?
bosh_pre_destroy.rb now has an execution timeout of 2sec. I can't reproduce the intermittent failure anymore with this change. It does seem these tests are slower though, probably because of Bundler.
reverted the commit which adds extra debugging to bosh_pre_destroy.rb for the problems we were having under Rspec in Travis.

:+1: Looks good to me now.  I'll leave this for a second reviewer to merge.

PS: I didn't link to the individual Mimic PRs because I didn't want this PR, where we no longer use Mimic, to be linked and affect the likelihood of them being merged.

I've not reviewed the coverage, but I have run the tests locally and they all pass on Ubuntu 14.04.

I've hit the timeout issue:
------------------------------
• Failure [2.008 seconds]
BoshPreDestroy
/Users/colin/Documents/Boulot/gds/git/paas-cf/concourse/scripts/bosh_pre_destroy_test.go:99
  no deployments
  /Users/colin/Documents/Boulot/gds/git/paas-cf/concourse/scripts/bosh_pre_destroy_test.go:79
    should return a zero exit code and no output [It]
    /Users/colin/Documents/Boulot/gds/git/paas-cf/concourse/scripts/bosh_pre_destroy_test.go:78

    Timed out after 2.000s.
    Expected process to exit.  It did not.

    /Users/colin/Documents/Boulot/gds/git/paas-cf/concourse/scripts/bosh_pre_destroy_test.go:75
------------------------------
•

Summarizing 1 Failure:

[Fail] BoshPreDestroy no deployments [It] should return a zero exit code and no output

The second time it worked fine. Can we increase it more?

The second time it worked fine. Can we increase it more?

Do you think 3s would be enough?
Out of interest, are you able to reproduce it with ginkgo -untilItFails?

ginkgo -untilItFails didn't make it fail even after 90 attempts (and 90 funny messages, thanks ginkgo!).
I have deleted my gemset, bundle installed again and ran the test with 3s timeout. It succeeded and took just above 2s: 2.099s. The second time it took only 0.950s.
I reckon 3s is fine.

Can we remove aruba from scripts/Gemfile? It doesn't look like it's used there.

Rebased with 3secs.

Can we remove aruba from scripts/Gemfile? It doesn't look like it's used there.

Do you mean..

paas-cf/scripts/Gemfile? It was removed in #386
paas-cf/concourse/scripts/Gemfile? It's still used by the other tests in concourse/scripts/spec/ at the moment

OK it's already in master, thanks.
Merging.


Not possible to log actual underlying Docker images

We use Binci in our CI/CD. At like a 30% random rate, services fail to start sometimes.
https://github.com/binci/binci/blob/master/src/services.js#L86 calls https://github.com/binci/binci/blob/master/src/proc.js#L13 but it does it with silent: true and no way to override it.
Maybe some debugging options for errors?

This is an issue I've been wavering on since I started this project. Typically (at least for myself and teams I've used Binci with) a failed service start is a matter of configuration.
Since under the hood Binci essentially just constructs a series of docker run ... commands then links them on the "primary" container, a service failure seems easiest to debug by simply running that service and/or tailing docker logs.
That being said, it definitely would be preferable to have some sort of output. I feel like it would be best to have this as a verbosity level control so info level stuff doesn't output unless explicitly requested, overriding the default silent: true for errors caught and spitting them out.

Duplicate of #87


Table server side procesing error on update from 3.0.12 to 3.1.1

Updating from coreui/vue v3.0.12 to v3.1.1 makes an unwanted behavior on column filters when usign external filter/sorter/pagination methods.
Summary of the problem.
The column filters disappear when using an external filter methods and no results returned from the server, forcing to use the cleareable attribute to eliminate all the filters, order, pagination, per_page option and current page selected that the user add to the table.
General description
I manage to use CDataTable component and CPagination component to work with server side procesing. This way the table only have the rows that the client wants to see and nothing more.
Description of behavior on v3.0.12
On v3.0.12 the CDataTable.vue component show the column filters depending on predefined arrays (the fields prop).
Image showing table without any filter. The table has only 5 rows and calculate the number of pages depending of total rows.

When i add a filter that iknow it will return 0 registers (integer id equals to a letter, id == "y" in this case). The server respond with 0 registers. In this case the column filters it still visible.

Line 83 of CDataTable.vue component was the only condition to determine if the filter input shows on the table.
v-if="!fields || fields[index].filter !== false"
Description of behavior on v3.1.1
The same situation as described before, but in this case cause there is no registers after filter, the table wont show the column filters, and even if the user make a mistake typing something, the user wil have to clear all the filters to fix his mistake.

On lines 101 and 102 of the new version it was added another condition to determine if any of the results has te column name.
v-if="itemsDataColumns.includes(colName) && (!fields || fields[index].filter !== false)"
Suggestion of correction
It would be better to add a custom option on the filter prop to allow user show filter column using the header column names instead of the first register of the table, for the moment i will use a temporary fix, adding a default row with a different index to prevent the table shows that temporary row.
I apologise if my english is not the best, and if the description of problem is not the best, this is my first time describing a problem.

I realize that instead of using that crazy idea that i have, i overwrite the ${columnName}-filter slot and create my custom column filter method.
<template :slot="field.key+'-filter'" v-for="field in fields.filter(f => f.filter)">
    <input
        @input="filterColumnChange(field.key, $event)"
        :placeholder="'Buscar '+field.label"
        class="form-control form-control-sm"
        :aria-label="`column name: '${field.key}' filter input`"
    >
</template>

This way all column filters shows always.


Webview2: Clicking a select element not popping up the drop down list

Steps to reproduce the bug

Create basic WinUI3.0 desktop app only embedding a Webview2 control
Set the source of the control to https://localhost:8080
Serve on https://localhost:8080 a basic index.html page with a select element
Start the app
Click the select element
Notice the dropdown element doesn't pop up. There's a slight flickering effect visible and the drop down closes again.

Expected behavior
Drop down list should pop up
Screenshots
This is screenshot from Edge browser

This is screenshot from the app

Version Info
Packaged WinUI3.0 Desktop app, WindowsAppSDK 8.0
NuGet package version:
WinUI 3 - Windows App SDK 0.8: 0.8.0

Windows app type:

UWP
Win32

Yes

Windows version
Saw the problem?

Insider Build (xxxxx)

October 2020 Update (19042)

May 2020 Update (19041)
Yes

November 2019 Update (18363)

May 2019 Update (18362)

October 2018 Update (17763)

April 2018 Update (17134)

Fall Creators Update (16299)

Creators Update (15063)

Device form factor
Saw the problem?

Desktop
Yes

Xbox

Surface Hub

IoT

Additional context
This works both in Edge and a Win32 C++ app (taken from here https://github.com/MicrosoftEdge/WebView2Samples/tree/master/SampleApps/WebView2APISample) so seems related to WinUI3.0. Hence I'm reporting it here instead of on the Webview2 feedback repo.

@codendone and @krschau FYI

Hi @koenvd, thanks for reporting this! I can repro it and will continue to investigate.

Nice @krschau!
Just came into mind this is probably related to the other issue I filled today: https://github.com/microsoft/microsoft-ui-xaml/issues/5615
Probably if you click the select item, the window will loose focus, which closes the drop down again. The linked ticket illustrates the issue for a button. But probably the same will happen for a select control.
Otherwise I give that a try tomorrow and will give another update.

Seems indeed the windows looses focus when clicking the select button:

Gist here: https://gist.github.com/koenvd/d1a4877e3189276da8b9a0b4d0dcc4d4
So this one might be fixed by #5615 .
Leaving it up to you if you want to keep both issues open.

This seems to be working fine as well in an UWP app with the backported Webview2 in the WinUI2.7 prerelease.
Just wondering if you have a rough idea when this would be fixed in WinUI3.0 @krschau @codendone ?

FYI but still seems to fail with WindowsAppSdk v1.0.0-preview1.

Also happening on 1.0.0-experimental1. Is anyone working on this? A web browser control that doesn't support Select isn't a whole lot of use..

Still happening on WindowsAppSdk v1.0.0-preview3 as well.

I'm not sure if it's related but we also have a major issue with capturing the mouse using navigator.xr.requestSession

the mouse pointer seems to be half captured, tracking runs at 50% movement/speed than it does in Edge or Chromium and the mouse pointer is periodically reset, like it's fighting something for focus.

This is it working in Edge:
https://ispyrtcdata.blob.core.windows.net/downloads/capture_edge.mp4
Not working in WebView2:
https://ispyrtcdata.blob.core.windows.net/downloads/capture_webview2.mp4

@krschau is this going to be addressed any time soon? For such a critical issue it's very concerning that nothing has been done about it for 3+ months - we're putting a large amount of effort into our app and WebView2 is a critical part of it and no  update to this ticket is quite a massive danger sign. Just need to know if/ when it will be addressed or if we need to re-tool.

@ispysoftware I'm sorry that this has taken so long. Unfortunately the fix won't make it to the 1.0 release, but it should be in 1.1. (Just so you know, we don't use the Assignee field in GitHub for most WinUI 3 items, but owners are assigned elsewhere.)

Thanks. Do you know when the 1.1 release is expected to ship?

@krschau is there any workaround for this issue? Our project will be released very soon, now this issue is a stop-shipping issue for us. We have no time to wait till 1.1 release is shipped. Please help!

This will be fixed in the next 0.8 and 1.0 previews, but I can't really give more detail than the roadmap on timing for those.

Fixed in Windows App SDK 1.0.1, check out the release notes for more info.

This issue is still occurring in WindowsAppSDK v 1.5.240404000.