Problem with aliases

For following code:
module Aaa {
  export class Abc {
  }
}
module Bbb {
  type aAbc = Aaa.Abc;
  export class Xyz {
    constructor() {
      var c: aAbc = new aAbc();
      var d: any;
      if (d instanceof aAbc) {

      }
    }
  }
}

I'm getting:
...\file1.ts(11,25): error TS2304: Build: Cannot find name 'aAbc'.
...\file1.ts(13,24): error TS2304: Build: Cannot find name 'aAbc'.
I can use alias when declaring variable, but in ither places it seems to be ignored.
Is this by design? (if so, why?)

type only creates a type name; the expression new aAbc() refers to a value.
If you want both, use import aAbc = Aaa.Abc instead. That will get both the type and the value.

OK, "new aAbc()" might refers to something else. but "aAbc" refers to "type aAbc = Aaa.Abc" and as per explanation "Type aliases are exactly the same as their original types; they are simply alternative names" this should work. In C# you can use that, so expectation is to have it working here too...
My understanding that if this is "alias" then it should work everywhere (like when before compilation using find&replace). Otherwise I do not understand reason of adding something like that to language...

A TypeScript class introduces a type name and a value of that name (the constructor function), as this is essentially how JavaScript works. Expecting something to just work the same as C# is not realistic, there are very fundamental differences between the two.

OK, but following your thoughts, the JS does not have "alias" thing. If you introduce new feature to language and use name for that feature that "collides" with existing feature of another language, you borrow also meaning of that name.
Honestly expecting to write:
var a: aAbc = new Aaa.Abc();
makes completely no sense from code readability.
Either remove those "semi aliases" or expand usage of them.

If you introduce new feature to language and use name for that feature that "collides" with existing feature of another language, you borrow also meaning of that name.
This is simply not the case. There are numerous examples of languages that have similar concepts with different meanings (generics in Java vs C#, differences in regular expression syntax between languages, symbols in ES6 vs Ruby, async in F# vs C#, etc). There are high level concepts that many languages want but whose specifics must work differently in each language for a variety of reasons. To expect a new language to use those concepts if and only if they can match exactly with some other completely different language is not realistic.
JavaScript and C# simply have different semantics for precisely what a class means. You cannot do this in C#:
class MyClass { ... }
// elsewhere:
var cons = MyClass;
var c = new cons();

The declaration of MyClass does not introduce a value called MyClass. And so C# gives you an error on this code like "MyClass is a 'type' but is used like a 'variable.' This is not how JavaScript works. JavaScript allows exactly this type of thing, which also means you need a way to differentiate between the type of the constructor function (the type of cons above) and the type of objects it creates (c above). Classes are fundamentally different in the two languages, and so you will see differences emerge around other features that interact with them. import aliases allow you to create aliases that represent multiple kinds of things (ex a name that is both a type and a value). Type aliases do not allow this, but do let you alias union types and other things to a single, easy to use name.

Hey @wgebczyk , i think you're misinterpreting this bit: "OK, "new aAbc()" might refers to something else. but "aAbc" refers to "type aAbc = Aaa.Abc" and as per explanation "Type aliases are exactly the same as their original types; they are simply alternative names" this should work."
The key part to note is " the same as their original types".   For something like class Abc there are actually two things created.  A value and a type.  A type alias only allows you to alias a type.  So, the quote up there is exactly correct.  The type alias is exactly the same as the original type.  But it is not the same as the original type AND value.
Type aliases give you an alias to a type.  And you can get a value alias with Javascript's built in 'var' feature.

OK. I agree that from JS perspective that might be resonable. At same time you agree that "var a: aAbc = new Aaa.Abc();" is weird - defines type "aAbc" and initialize with "Aaa.Abc". (Lets skip for moment type inference as we can split this statement into 2 of them declaration and initialization). Readability of such constructs is lowered.
The question is - why we cannot expand aliases to be used in other places? The TypeScript team owns specification (grammar) and can allow usage only in those areas that will not introduce some possible conflicts?

At same time you agree that "var a: aAbc = new Aaa.Abc();" is weird

It isn't. TypeScript is not C#. There is not a 1:1 correlation between things you can new and types.

why we cannot expand aliases to be used in other places?

There is already syntax for doing what you want. It's import aAbc = Aaa.Abc;. We don't need two ways of doing the same thing.
We have a very simple system here:

You want the value: use var
You want the type: use type
You want both: use import

This is like asking why we don't expand the definition of "unicycle" to include objects with two wheels. There's already a word for that, "bicycle", and the prefixes uni- and bi- have meanings.

OK. You are right. The import would be sufficient and lets put alias for some obscure usages :)
Thank you!

@RyanCavanaugh I have searched for explanation and found it here.
The explanation should exist in the wiki.
For instance
https://github.com/Microsoft/TypeScript/wiki/Modules
It is written: ". This is similar to using var, but also works on the type and namespace meanings of the imported symbol"
It should be added also "similar to 'type' but, but also works on the value.".


small fix in adapter_idb, causing callback to be missing

I was getting a stack of errors (below) related to a missing callback, when running your couch port in the browser.
Although I don't think browser usage was your intent I wanted to see if I could get it working.
That said, I'm also importing this modified version of your port for a node-side instance of couch and it seems to be ok.
From what I could see, declaring this as an arg in the function being run via call was causing the arguments to shift, so that the value of this was the options and  the value of opts was the callback (for me anyways).
This change seems to have unblocked things and I can now see the db created in the "browser storage inspector" (firefox).
The error(s):
Uncaught TypeError: fun is not a function
tryCode pouchdb-adapter-idb.js:786
runCallback pouchdb-adapter-idb.js:801
completeSetup pouchdb-adapter-idb.js:1511
oncomplete pouchdb-adapter-idb.js:1543
onsuccess pouchdb-adapter-idb.js:1541
init pouchdb-adapter-idb.js:1476
IdbPouch pouchdb-adapter-idb.js:969
runAction pouchdb-adapter-idb.js:800
applyNext pouchdb-adapter-idb.js:796
enqueueTask pouchdb-adapter-idb.js:808
IdbPouch pouchdb-adapter-idb.js:968
idb mod.ts:41
PouchDB pouchdb-core.js:1401

Were you using deno bundle to compile for browser? How would I reproduce?
this args in the first position should be stripped out when TypeScript compiles to JavaScript. If it remained, then there's a bug likely to be filed against either Deno or the TypeScript compiler.

I'm running it inside Alephjs.
https://github.com/alephjs/aleph.js
I used their basic...
deno run -A<EMAIL_ADDRESS>init --template react
...as a starting point
I  created a route called test.tsx with...
let db: PouchDB.Database<{}> | undefined;

if (window.document) {
  console.log("try COUCH");
  db = new PouchDB("resumes", { adapter: "idb"});
} else {
  db = undefined
}

@elverskog I'm sorry. While I can reproduce your issue, the problem is with Aleph, not with this bootstrap code.
TypeScript explicitly handles this passed as the first parameter as a special case, so that the type of this in a function body can be declared. The TypeScript compiler and the Deno bundler both correctly strip this keyword out.
Aleph's bundler does not do this (pun intended), it instead minifies the type parameter to a real JavaScript parameter.
Considering the fact that Aleph is in early stages and considered alpha software, I would recommend filing a bug with them. If they cannot fix the issue in a timely manner, I'm not sure what the outcome should be. A lot of software written in TypeScript takes advantage of this feature (including Deno's standard modules), so I'm currently leaning towards "won't fix" in this codebase.

@aaronhuggins Ah. Super interesting.
Thanks for taking a look.

@elverskog I'm willing to continue the conversation and revisit this PR depending on what Aleph's response is. Please feel free to @ me on the issue with Aleph.js as I'd like to follow it.

Added the issue. Hopefully I explained it well enough.
@'d you but just in case https://github.com/alephjs/aleph.js/issues/485

This is a different topic but I noticed indexedDB in the build.
I also noticed a sqlite file created with a name matching my server-side db.
Is pouch falling back to sqlite and is that what i should expect inside deno's runtime?
I assume levelDB is not supported in deno...or is maybe not preferred?
If deno supported levelDB (does it?) I assume porting pouchdb-node (instead) would make sense.
So just curious what is happening now.
Is the couch code failing to find indexed DB and therefore
Just curious: Since your goal was to support server-side pouch wouldn't "porting" pouchdb-node (w/ pouchdb-adapter-leveldb)
I see the idb module in there, so I am confused.

@elverskog The answers you seek are probably here. If there's not enough detail concerning the decisions I made porting PouchDB or you'd like more details not covered in the README, ping me again. :smiley:


Proposal for collaboration

Hi, my name is Aleksei. About me.
I'm very curious in General Artificial intelligence researches and follow OpenCog projects and events.
I’m sure that OpenCog team is doing great job to develop general AI and I would be happy to help.
I have ideas how to develop Artificial Intelligence as a child from birth to youth and I believe that Hyperon is the best framework for trying to implement it.  Details here
I’ve created a simplified Python example of sensory perception exercise for child (Catching the toy from the hands of an adult) in  child_ai.py file.
To use Hyperon  Space / Atoms functionality I’ve created several functions in Rust (source code in graph_kb.rs file). They duplicate similar functions in Python child_ai.py and have the same names. At the moment there is no integration between Python and Rust for these functions.
I understand that general AI is a very long way and there are many another ideas.
I don’t insist to implement my idea and ready to help with other tasks that you think have more potential and priority.
Now I’m looking for new opportunities as my employer SAP winds down business in Russia. If you have any open position I would be happy to apply and support your projects in any format.
Looking forward for your feedback.
Best regards, Aleksei

@h555yand , please, consider raising this PR in https://github.com/trueagi-io/metta-examples/

Hi Aleksei, thank you for your interest and for sharing your code.
As far as I understand you have implemented the idea in Python and a Rust part is to demonstrate how it could be implemented in the core library. And as far as I see the Python code don't use Metta, so it is an independent implementation.
I would also suggest raising PR to the https://github.com/trueagi-io/metta-examples/ and then it can be discussed how your idea can be implemented using Metta or what can be added to Metta to make implementation possible.

Now I’m looking for new opportunities as my employer SAP winds down business in Russia. If you have any open position I would be happy to apply and support your projects in any format.

I will write you an email on this.

Hi Vitaly, thank you very much for the quick response.
Your understanding is correct. Python code is independent at the moment and Rust functions are commented and not used there.
OK, I will create a new PR in https://github.com/trueagi-io/metta-examples/
Best regards, Aleksei

Hi Alexey, Vitaly,
As agreed I’ve made updates for Expression Atoms for Graph knowledge base task.
The task is to demonstrate possible approaches how to use Graph knowledge base data with MeTTa.
Typical  Graph knowledge base consists of Edges with Nodes, and they can have features to store information of different types (sparse distributed vectors for code and weights, IDs, Names, Types etc).
Use case is to enter Edges with Nodes in MeTTa and then parse / interpret them in Graph knowledge base format that used in Child_AI idea https://github.com/h555yand/hyperon-experimental/blob/main/python/tests/child_ai/README.md
For Edges with Nodes the expression is used “((LinksEdge Link1) (MemoryNode Node1) (MemoryNode Node2))”.
This is similar to OpenCog classic format expression "(EvaluationLink(PredicateNode eats) (ListLink (ConceptNode Sam)(ConceptNode flies)))"
In future we can change it to Type format like (: LINK Type).
I’ve modified the Rust file graph_kb.rs and have not changed any Python code. As I understand there are parse / interpret functions that are in the process of transferring from Python into Rust.
I’ve created several new functions ( GraphKB::parse_graph and others) in addition to the existing SexprParser / interpreter::interpret functions. In future, if we decide to include such functionality in MeTTa, we need to adapt the existing functions.
The existing Rust functions parse / interpret the expression “((LinksEdge Link1) (MemoryNode Node1) (MemoryNode Node2))” as Expression Atom with children Symbol Atoms. To use it with Graph knowledge base we need to convert them into Expression Atom with children Grounded Atoms (Link Edge, Source and Destination Nodes) .
The first step is to try to find Grounded Atoms by Symbol Link and Node Names. If no success, we create new edge and nodes, using Name from MeTTa Expression and default values for other fields.
If no need in interpretation we just add Expression Atom in Space.
To use interpretation we need to add special functions to work with Graph, similar to existing functions in Python self.add_atom(r"get-type", newGetAtomTypeAtom(self)).
We can return to MeTTa requested  information from Grounded Atoms in addition to Symbol Names and execute requested actions (to find linked nodes for example).
There are two test cases: (1) when Grounded Atoms (Link1, Node1, Node2) exist,
(2) when Grounded Atoms (Link11, Node11, Node22) don't exist.
Best regards, Aleksei.

Hi Alexei,
When I was talking about more tight integration with Metta I thought about something like the following program:
(bind! &memory (new-space))

(= (predict-next obj)
    (select-max-weight
        (collapse (match &memory (next-link obj $next $w) ($w $next)))
    )
)

(= (add-next-observation $prev $next)
    (case (match &memory (next-link $prev $next $w) $w)
        (%void% (call:add &memory (next-link $prev $next initial-weight)))
        ($w (call:replace &memory
            (next-link $prev $next $w)
            (next-link $prev $next (increment-weight $w))))
    )
)

!(add-next-observation "a" "b") ; adds (next-link "a" "b" <initial-weight>) into &memory kb
!(add-next-observation "a" "a") ; adds (next-link "a" "a" <initial-weight>) into &memory kb
!(add-next-observation "a" "b") ; increments weight of the (next-link "a" "b" <initial-weight>)
!(add-next-observation "a" "b") ; increments weight of the (next-link "a" "b" <initial-weight>)

!(predict-next "a") ; returns prediction with a maximum weight "b"

I wrote it to demonstrate how one can implement prediction of the next object (one of the examples I found in README.md file you provided). Basically we have &memory database which contains information about next objects and weights. Each possible next object memoized from experience is represented by (next-link <prev-object> <next-object> <weight>) where next-link is a symbol, <prev-object> and <next-object> are some atoms, and <weight> is grounded atom which contains instance of numpy array or any other convenient low-level structure.
I have added predict-next and add-next-observation functions to demonstrate how one can implement such functionality in Metta. Of cause operations on grounded weights cannot be implemented in pure Metta so one need implement few grounded functions in addition:

select-max-weight which gets list of pairs (<weight> <object>) and returns the object with maximum weight;
initial-weight which is a symbol to assign initial weight to the link;
increment-weight which is a function to increment weight when some instance of next link is created from experience.

The purpose of the example is to demonstrate how one of your examples could be implemented in Metta like style with minimal necessary grounded objects and operations. Here only operations on weights should be implemented manually. All other functions (bind!, new-space, match, call, case, collapse) are part of the Metta library. And operations on weights can be implemented even in Python.
I am not sure actually how far this example from what you have implemented in the PR though. May be it is simpler to discuss it on a call and then continue discussing here.

One can also write this code in pure Python if it is more convenient like code below. But still there is no need to create custom Link and Node instance, one can use built-in Metta expressions and symbols:
from hyperon import *

memory = GroundingSpace()

class Weight:

    def __init__(self):
        self.w = 1

    def __lt__(self, other):
        return self.w < other.w

    def inc(self):
        self.w += 1

def next_link(prev, next, weight):
    return E(S('next-link'), prev, next, weight)

def predict_next(obj):
    candidates = memory.subst(next_link(obj, V('next'), V('w')), E(V('w'), V('next')))
    if len(candidates) == 0:
        return None
    max_weight_expr = max(candidates,
                          key=lambda e: e.get_children()[0].get_object().value)
    return max_weight_expr.get_children()[1]

def add_next_observation(prev, next):
    existing = memory.query(next_link(prev, next, V('w')))
    if len(existing) == 0:
        memory.add_atom(next_link(prev, next, ValueAtom(Weight())))
    else:
        assert(len(existing) == 1)
        weight = existing[0]['w']
        new_weight = weight.get_object().value
        new_weight.inc()
        memory.replace_atom(next_link(prev, next, weight), next_link(prev, next, ValueAtom(new_weight)))

add_next_observation(S('a'), S('b')) # adds (next-link "a" "b" <initial-weight>) into &memory kb
add_next_observation(S('a'), S('a')) # adds (next-link "a" "a" <initial-weight>) into &memory kb
add_next_observation(S('a'), S('b')) # increments weight of the (next-link "a" "b" <initial-weight>)
add_next_observation(S('a'), S('b')) # increments weight of the (next-link "a" "b" <initial-weight>)

print(predict_next(S('a'))) # prints prediction with a maximum weight "b"

@h555yand as we discussed I closing this, please move this PR into https://github.com/trueagi-io/metta-examples/.
It makes sense to write a small README.md with instructions how to run this project before hyperon library is published.


DX-070 | Fix Grid Height

The height function of the datagrid has been deprecated, and due to this, we removed the height function. The problem is that we implemented the height function to get around an issue that we were experiencing with virtual/infinite scolling and fixed columns. When data took too long to load, the grid would misalign, and would not sort correclty. This was mainly due to the fact that DevExpress had changed the way that it handled virtual/infinite scrolling.
With the hack that we used, we were experiencing strange behaviour in v19, but that behaviour got worse with later versions, when DevExpress replaced the scrolling engine all together. https://supportcenter.devexpress.com/ticket/details/t1034020/datagrid-treelist-virtual-scrolling-engine-enhancement
I have tried removing the height function, setting the scrolling to legacyMode, forcing height refreshes, and changing event handlers. All of which failed to achieve the correct results.
I eventually ran across: https://supportcenter.devexpress.com/ticket/details/t1046212/datagrid-virtual-scrolling-gray-boxes-are-shown-in-fixed-columns-when-the-grid-is
After setting the prerenderedRowCount in the onInitialized event handler, and setting the height to a css value, it worked.
We should still monitor this, but it seems to have fixed all of the grid issues we were having.

Just to clarify - no other changes are required in services using this gem, just a simple upgrade ?

Yip. Just an upgrade of the gem and npm packages.


RuntimeError: alpha must be constant for bounded relu while bounded relu is not used.

When I'm trying to run this model:
{
  "name": "Function_0",
  "ops": [
    {
      "element_type": "float",
      "inputs": [],
      "name": "Parameter_0",
      "op": "Parameter",
      "outputs": [
        "Parameter_0_0"
      ],
      "shape": [
        3,
        4,
        5
      ]
    },
    {
      "element_type": "float",
      "inputs": [],
      "name": "Constant_5",
      "op": "Constant",
      "outputs": [
        "Constant_5_0"
      ],
      "shape": [],
      "value": [
        "3.40282e+38"
      ]
    },
    {
      "element_type": "float",
      "inputs": [],
      "name": "Constant_1",
      "op": "Constant",
      "outputs": [
        "Constant_1_0"
      ],
      "shape": [],
      "value": [
        "0"
      ]
    },
    {
      "axes": [],
      "inputs": [
        "Parameter_0"
      ],
      "name": "Broadcast_2",
      "op": "Broadcast",
      "outputs": [
        "Broadcast_2_0"
      ],
      "shape": [
        3,
        4,
        5
      ]
    },
    {
      "axes": [
        0,
        1,
        2
      ],
      "inputs": [
        "Constant_5"
      ],
      "name": "Broadcast_7",
      "op": "Broadcast",
      "outputs": [
        "Broadcast_7_0"
      ],
      "shape": [
        3,
        4,
        5
      ]
    },
    {
      "axes": [
        0,
        1,
        2
      ],
      "inputs": [
        "Constant_1"
      ],
      "name": "Broadcast_3",
      "op": "Broadcast",
      "outputs": [
        "Broadcast_3_0"
      ],
      "shape": [
        3,
        4,
        5
      ]
    },
    {
      "inputs": [
        "Broadcast_2",
        "Broadcast_3"
      ],
      "name": "Maximum_4",
      "op": "Maximum",
      "outputs": [
        "Maximum_4_0"
      ]
    },
    {
      "axes": [],
      "inputs": [
        "Maximum_4"
      ],
      "name": "Broadcast_6",
      "op": "Broadcast",
      "outputs": [
        "Broadcast_6_0"
      ],
      "shape": [
        3,
        4,
        5
      ]
    },
    {
      "inputs": [
        "Broadcast_6",
        "Broadcast_7"
      ],
      "name": "Minimum_8",
      "op": "Minimum",
      "outputs": [
        "Minimum_8_0"
      ]
    },
    {
      "inputs": [
        "Minimum_8"
      ],
      "name": "Result_9",
      "op": "Result",
      "outputs": [
        "Result_9_0"
      ]
    }
  ],
  "parameters": [
    "Parameter_0"
  ],
  "result": [
    "Result_9"
  ]
}

I get this error: RuntimeError: alpha must be constant for bounded relu
But i dont use bounded relu here :disappointed:
When Constant_1 value will be changed from 0 to any other value everything works.
there are the python scripts used to reproduce error:
Not working:
import ngraph as ng
import numpy as np

data = ng.parameter([2,3,4])
data_elem_dtype = np.float32

max_value = np.finfo(data_elem_dtype).max
min_value = 0

model = ng.minimum(ng.maximum(data, ng.constant(min_value, data_elem_dtype)), ng.constant(max_value, data_elem_dtype))

rt = ng.runtime()
comp = rt.computation(model, data)
print(comp.serialize(2))
comp(np.arange(24).reshape(2, 3, 4))

Working:
import ngraph as ng
import numpy as np

data = ng.parameter([2,3,4])
data_elem_dtype = np.float32

max_value = np.finfo(data_elem_dtype).max
min_value = 1

model = ng.minimum(ng.maximum(data, ng.constant(min_value, data_elem_dtype)), ng.constant(max_value, data_elem_dtype))

rt = ng.runtime()
comp = rt.computation(model, data)
print(comp.serialize(2))
comp(np.arange(24).reshape(2, 3, 4))

@tsocha can you try this branch https://github.com/NervanaSystems/ngraph/tree/pruthvi/fix_bounded_relu. I made a fix here, I am getting json parsing error with the serialized graph u attached in the issue.
terminate called after throwing an instance of 'nlohmann::detail::type_error'
  what():  [json.exception.type_error.304] cannot use at() with string
Aborted

I am hoping the above branch should fix the issue. if not, please attach a serialized_graph file. so I can reproduce my end.

This branch fixed the problem thanks a lot! :)


Differentiate between tabs and spaces (This is a request, not sure if possible)

I like the guide-size=1 look with spaces, but unfortunately this does not work when using tabs.
I have list set so my tabs do show up (So I can tell that some whitespace is made of tabs or not), I just would like to have some way of telling the indent-guides plugin to color tab based indentation differently from space-based indentation. For instance since only one character out of 4 will be highlighted with space-indentation the color can stand to be more prominent. However with tabs they are always fully colored so I'd like to configure a less prominent color for when they are tabs.

Check out https://github.com/thaerkh/vim-indentguides.


inconsistencies resolving .^ in an app vs a generator

Not sure if this is the right place for this ...
I'm butting my head trying to resolve a .^ inside an app, when the same twig works in a generator.
In :dojo:
> .^(json %cx /=home/348/web/easy/json)
>> [%camping %c %x [p=~wolsyp-filtuc-havrut-londys--pinbel-ballut-fodmyl-litmer q=%home r=[%ud p=348]] s=/json/easy/web]
... :: json object printed

When I evaluate that twig in a generator, I see the same [%camping ...] printf, but inside an app, it just prints the path and blocks:
/cx/~wolsyp-filtuc-havrut-londys--pinbel-ballut-fodmyl-litmer/home/348/web/easy/json

(same behavior in the ++prep and ++poke arms).
While I have ~1000 other questions about .^ (which is pretty lightly documented), this is the obstacle I haven't been able to work around.

Seemingly relevant tidbit:
In :dojo:
> .^(hart %e /(scot %p our)/host/real)
[p=%.y q=~ r=[%.y p=<|org urbit wolsyp-filtuc-havrut-londys--pinbel-ballut-fodmyl-litmer|>]]

Whereas in :talk, the same twig just produces the path:
/e/~wolsyp-filtuc-havrut-londys--pinbel-ballut-fodmyl-litmer/host/real

Wise question, glasshoppa!
The answer: blocking .^ is a very non-fundamental behavior.  It is
obviously not part of Nock proper (it turns into Nock instruction 11, which
is only implemented by the virtual extended Nock ++mock).  It is not even
available to %gall apps, which can only do non-blocking .^.  For the
programmer, the first layer blocking .^ works in is anything run by :dojo
(generators, command line, etc).
Internally, what you're doing with .^ from the dojo, and what you're doing
from a %gall app, are different.  From a %gall app, .^ will just use the
Arvo kernel namespace function ++scry on your behalf.  ++scry obviously has
no way to block.  So in Unix terms, it's essentially producing EWOULDBLOCK.
From the dojo, .^ is actually invoking the build vane, %ford.  The
superpowers of %ford are tremendous (and not well documented).  You're
essentially saying: start this build and call me back when it's done.  If
%ford does not actually have to block, it will give a result in the same
top-level Arvo step.  But it's quite prepared to block.
(Moreover, the way we implement "blocking" is to... abort the computation
and rerun it when we have the result we need.  With proper caching this is
surprisingly not-bad.)
If you want to do the same thing in :talk or another app, you need to
invoke %ford asynchronously the way :dojo does.  Sorry, there is no such
thing as "real" app blocking.  I can defend this: one of the things about
Urbit's super-event-centric I/O model is that your app state is always just
a noun, never some kind of saved execution context.  The illusion of
blocking is pushed up to the highest possible level.  The result of this is
that it makes sense to, for example, hotpatch a blocked :dojo.  There is no
"please exit the program to continue upgrading" or whatever.
On Fri, Jul 1, 2016 at 8:59 AM, Joe Bryan<EMAIL_ADDRESS>wrote:

Seemingly relevant tidbit:
In :dojo:

.^(hart %e /(scot %p our)/host/real)
[p=%.y q=~ r=[%.y p=<|org urbit wolsyp-filtuc-havrut-londys--pinbel-ballut-fodmyl-litmer|>]]

Whereas in :talk, the same twig just produces the path:
/e/~wolsyp-filtuc-havrut-londys--pinbel-ballut-fodmyl-litmer/host/real
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/urbit/arvo/issues/191#issuecomment-229984009, or mute
the thread
https://github.com/notifications/unsubscribe/AALyAcJuuBVUgYMiGvAfoh1GdcrdLOrAks5qRTl2gaJpZM4JDQQ8
.

Yeah, except what he's scrying exists, and shouldn't fail in any way.
Yours,
r

He's scrying into %eyre, so maybe that's a question for Anton!
On Fri, Jul 1, 2016 at 10:11 AM, Raymond Pasco<EMAIL_ADDRESS>wrote:

Yeah, except what he's scrying exists, and shouldn't fail in any way.
Yours,
r
--
You received this message because you are subscribed to the Google Groups "urbit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to<EMAIL_ADDRESS>To post to this group, send email to<EMAIL_ADDRESS>For more options, visit https://groups.google.com/d/optout.

The original example was a clay scry.
Yours,
r

By "in talk", do you mean in a talk eval, or in the talk app?
In the talk app, I would expect the eyre scry to work, but not the clay
one: clay scry into the past is also blocking due to marks needing to be
rewound.
On Friday, 1 July 2016, cgyarvin<EMAIL_ADDRESS>wrote:

He's scrying into %eyre, so maybe that's a question for Anton!
On Fri, Jul 1, 2016 at 10:11 AM, Raymond Pasco<EMAIL_ADDRESS><javascript:_e(%7B%7D,'cvml','ray@the.ug');>> wrote:

Yeah, except what he's scrying exists, and shouldn't fail in any way.
Yours,
r
--
You received this message because you are subscribed to the Google
Groups "urbit" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to<EMAIL_ADDRESS><javascript:_e(%7B%7D,'cvml','urbit-dev%2Bunsubscribe@googlegroups.com');>
.
To post to this group, send email to<EMAIL_ADDRESS><javascript:_e(%7B%7D,'cvml','urbit-dev@googlegroups.com');>.
For more options, visit https://groups.google.com/d/optout.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/urbit/arvo/issues/191#issuecomment-230000160, or mute
the thread
https://github.com/notifications/unsubscribe/ABxXhsUsqXKToTCA-EK25dDVxFA25Dmgks5qRUp3gaJpZM4JDQQ8
.

I mean a :talk eval... I just added that example since it's a similar
test-case, and demonstrates the same dichotomy.
Thanks all for the details! My most pressing follow-up: how do I resolve a
non-blocking :wish? Is there some type of explicit callback mechanism, or
do I have to produce (and then respond to) moves?
On Fri, Jul 1, 2016 at 1:14 PM, Anton Dyudin<EMAIL_ADDRESS>wrote:

By "in talk", do you mean in a talk eval, or in the talk app?
In the talk app, I would expect the eyre scry to work, but not the clay
one: clay scry into the past is also blocking due to marks needing to be
rewound.
On Friday, 1 July 2016, cgyarvin<EMAIL_ADDRESS>wrote:

He's scrying into %eyre, so maybe that's a question for Anton!
On Fri, Jul 1, 2016 at 10:11 AM, Raymond Pasco<EMAIL_ADDRESS><javascript:_e(%7B%7D,'cvml','ray@the.ug');>> wrote:

Yeah, except what he's scrying exists, and shouldn't fail in any way.
Yours,
r
--
You received this message because you are subscribed to the Google
Groups "urbit" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to<EMAIL_ADDRESS><EMAIL_ADDRESS>');>
.
To post to this group, send email to<EMAIL_ADDRESS><javascript:_e(%7B%7D,'cvml','urbit-dev@googlegroups.com');>.
For more options, visit https://groups.google.com/d/optout.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/urbit/arvo/issues/191#issuecomment-230000160, or
mute
the thread
<
https://github.com/notifications/unsubscribe/ABxXhsUsqXKToTCA-EK25dDVxFA25Dmgks5qRUp3gaJpZM4JDQQ8

.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/urbit/arvo/issues/191#issuecomment-230000633, or mute
the thread
https://github.com/notifications/unsubscribe/AAq3YM_aGefCAkL06Ak_j0AJqi2Xrjp5ks5qRUr2gaJpZM4JDQQ8
.

In practice, the flow for an event request versus a ++scry is totally
different.  So if a specific request succeeds on one path rather than
another, the place to look is that path.  They should match...
On Fri, Jul 1, 2016 at 10:13 AM, Raymond Pasco<EMAIL_ADDRESS>wrote:

The original example was a clay scry.
Yours,
r

Yep, you gotta do it with moves.
On Fri, Jul 1, 2016 at 10:17 AM, Joseph Bryan<EMAIL_ADDRESS>wrote:

I mean a :talk eval... I just added that example since it's a similar
test-case, and demonstrates the same dichotomy.
Thanks all for the details! My most pressing follow-up: how do I resolve a
non-blocking :wish? Is there some type of explicit callback mechanism, or
do I have to produce (and then respond to) moves?
On Fri, Jul 1, 2016 at 1:14 PM, Anton Dyudin<EMAIL_ADDRESS>wrote:

By "in talk", do you mean in a talk eval, or in the talk app?
In the talk app, I would expect the eyre scry to work, but not the clay
one: clay scry into the past is also blocking due to marks needing to be
rewound.
On Friday, 1 July 2016, cgyarvin<EMAIL_ADDRESS>wrote:

He's scrying into %eyre, so maybe that's a question for Anton!
On Fri, Jul 1, 2016 at 10:11 AM, Raymond Pasco<EMAIL_ADDRESS><javascript:_e(%7B%7D,'cvml','ray@the.ug');>> wrote:

Yeah, except what he's scrying exists, and shouldn't fail in any
way.
Yours,
r
--
You received this message because you are subscribed to the Google
Groups "urbit" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to<EMAIL_ADDRESS><EMAIL_ADDRESS>');>
.
To post to this group, send email to<EMAIL_ADDRESS><javascript:_e(%7B%7D,'cvml','urbit-dev@googlegroups.com');>.
For more options, visit https://groups.google.com/d/optout.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/urbit/arvo/issues/191#issuecomment-230000160, or
mute
the thread
<
https://github.com/notifications/unsubscribe/ABxXhsUsqXKToTCA-EK25dDVxFA25Dmgks5qRUp3gaJpZM4JDQQ8

.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/urbit/arvo/issues/191#issuecomment-230000633, or mute
the thread
https://github.com/notifications/unsubscribe/AAq3YM_aGefCAkL06Ak_j0AJqi2Xrjp5ks5qRUr2gaJpZM4JDQQ8
.

--
You received this message because you are subscribed to the Google Groups
"urbit" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to<EMAIL_ADDRESS>To post to this group, send email to<EMAIL_ADDRESS>For more options, visit https://groups.google.com/d/optout.

I don't know what happened on that email thread, but your response became "my" comment here ... ;)
My original twig is .^(json %cx /=home/348/web/easy/json); what kind of move does that translate into?

Sounds like a bug!

Ford turns it into a clay subscription, %warp. It's probably easiest to
send the request to ford as a twig; you'll have to unwrap the vase somehow,
but that's the case either way.
On Friday, 1 July 2016, Raymond Pasco<EMAIL_ADDRESS>wrote:

Sounds like a bug!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/urbit/arvo/issues/191#issuecomment-230003161, or mute
the thread
https://github.com/notifications/unsubscribe/ABxXhi0H8MG7jTdkVyaBfyoTP2j5ia6rks5qRU2igaJpZM4JDQQ8
.

This all makes sense, conceptually, but I'm having some trouble making it concrete  Can you point towards any examples of that pattern?
In more detail, a ++move is a {bone term wire *}, at least as taught in the Arvo tutorial (elsewhere, it's a bunch of other things ;)).
Here's a basic recursive move in an app -- it pokes the current app with an atom of 1. (ost, our, and dap are in the subject via bowl):
[[ost %poke /foobar [our dap] %atom 1] ~]
If I'm reading that correctly ost is the bone, %poke is the term, and [/foobar [our dap] is the wire, while %atom is references a mark and 1 is the arbitrary argument to our move.
What should I change to create a %clay %warp subscription, ie, what should the term and wire be?

Something like [ost %warp /foobar [our our] %home `[%sing %x ud+134
/web/easy/json]]~, looking at lib/kiln.hoon and arvo/zuse.hoon
On Friday, 1 July 2016, Joe Bryan<EMAIL_ADDRESS>wrote:

This all makes sense, conceptually, but I'm having some trouble making it
concrete Can you point towards any examples of that pattern?
In more detail, a ++move is a {bone term wire *}, at least as taught in
the Arvo tutorial (elsewhere, it's a bunch of other things ;)).
Here's a basic recursive move in an app -- it pokes the current app with
an atom of 1. (ost, our, and dap are in the subject via bowl):
[[ost %poke /foobar [our dap] %atom 1] ~]
If I'm reading that correctly ost is the bone, %poke is the term, and [/foobar
[our dap] is the wire, while %atom is references a mark and 1 is the
arbitrary argument to our move.
What should I change to create a %clay %warp subscription, ie, what should
the term and wire be?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/urbit/arvo/issues/191#issuecomment-230008336, or mute
the thread
https://github.com/notifications/unsubscribe/ABxXhubUEKFEBFRWFR1e-Fk-uHrxD0btks5qRVMUgaJpZM4JDQQ8
.

Hm, the email reply authors seem to be getting somewhat scrambled.

It's a creative take on pseudonymity ...

Github is always right, y'all are just crazy.
On Fri, Jul 1, 2016 at 11:26 AM, Joe Bryan<EMAIL_ADDRESS>wrote:

It's a creative take on pseudonymity ...
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/urbit/arvo/issues/191#issuecomment-230016506, or mute
the thread
https://github.com/notifications/unsubscribe/AFS8GLXSeMD6f7luNl90IbqWjWVmcg42ks5qRVvegaJpZM4JDQQ8
.

In your example, /foobar is the wire, and [[our dap] %atom 1] is the * part of the move.
For a clay subscription, %warp is your term, and the wire can be anything you want.  The rest of the move should be a sock and a riff, as you can see by searching for $warp in arvo/zuse.hoon.  sock is a pair of our ship and the other ship (also our ship in this case), and riff is the request itself.

I get [%swim-lost %c %warp] from your example. I'll dig into zuse and kiln and try to pull this apart. Thanks!

I've been meaning to investigate this further, so I haven't commented back... I never did get the file-load moves to work (trying only with raw moves ... ++foal and company are probably easier).
However, the advice here is wrong.
You can use .^ in an app, you just can't use the beak placeholders /===/. You have to use an explicit, "fully qualified path". See :talk: https://github.com/urbit/arvo/blob/master/app/talk.hoon#L2317.
ie, this common pattern:
/(scot %p our)/<explicit-desk>/(scot %da now)/...

I suspect that .^ will indeed work in an app as long as it doesn't have to block for any reason, just like ++sky functions in vanes.  For clay, that usually means local ship, current version.


Add support for DockerClient event streaming

I noticed that event streaming from the DockerClient as specified in the docker-java wiki (https://github.com/docker-java/docker-java/wiki#handle-events) doesn't work, as get(TypeReference typeReference, ResultCallback resultCallback) is not implemented in OkHttpInvocationBuilder.java

Thanks for the review @bsideup, I think I've addressed all of your comments


Add link to take contract quiz

Open to feedback on wording

📝 Changed routes:

/contracts

(view full size)

Commit 94193fcd0760dc96cefc88a7b9ab1c0312e57600 (https://thirdweb-67onyrye1-thirdweb.vercel.app).


Can't docker compose up in giantswarm/docs on M1 Mac because of missing ARM image

Description
Running docker compose up in https://github.com/giantswarm/docs/ results in an error from Docker because some images are missing the ARM version.
[09:55:51] ➜  docs git:(main) ✗ docker compose up --quiet-pull
WARN[0000] The "HUBSPOT_API_KEY" variable is not set. Defaulting to a blank string.
[+] Running 0/6
 ⠏ indexer-blog Pulling                                                             2.9s
 ⠏ docs-app Pulling                                                                 2.9s
 ⠏ proxy Pulling                                                                    2.9s
 ⠏ indexer-docs Pulling                                                             2.9s
 ⠏ api-spec-app Pulling                                                             2.9s
 ⠏ sitesearch-app Pulling                                                           2.9s
no matching manifest for linux/arm64/v8 in the manifest list entries

TODO

I think images should be made "ARM-compatible". I don't know how much the effort is, but I believe that with a lot of people getting M1 Macs, this is going to become needed sooner or later.

@LolloneS this issue is in no projects. So maybe it fell through the cracks.
/cc @anvddriesch @marians

I'd like to understand whether it's really necessary to have ARM images for M1 macs. Since the image build process relies pretty much on our common tooling (architect-orb), I can't even tell how easy it would be to build for ARM in addition.
I glanced over https://docs.docker.com/desktop/mac/apple-silicon/ and it looks like with Rosetta you can also run x64 images. Also it is mentioned that you should use docker-compose v2 at least for that.

Marcus tested on M1 and found it to work when adding platform: linux/amd64 to the elasticsearch service. Like this:
sitesearch-app:
    image: elasticsearch:6.8.6
    platform: linux/amd64

@LolloneS If you could check whether that works for you, that would be great.

The platform: linux/amd64 parameter works for me too. Thank you all for taking care of this!


Documentation for usage with Karma/Angular 4+

I assume that since older versions of mocha worked with older versions of Angular, that this is still possible, but all of the references I can find online for this are varied and not helpful. I think it would be helpful for this project if there was an official documentation for how to set up mocha for Karma and Angular.

the wiki would make sense for this... but we'd need someone who's familiar with Angular to create it.

@weswhite I'm not an owner of this project, but at a glance, this seems like a reasonable example to base it on:
https://github.com/shellscape/mocha-chrome
I greatly appreciate it by the way.

@weswhite yes, whatever is helpful for @kbirger would likely be helpful for the greater community.  I can't say what's needed because I don't use Mocha with Karma & Angular (though Mocha uses Karma to test itself)

@boneskull I figured out how to do this. Where should I put the steps? First open source contribution, sorry...

@weswhite 	Create a new wiki page.  make sure you link to it from the list of tutorials on the front wiki page.  thanks for your help!! :heart:

how is this going?.. i would really love to see how to make angular tests using mocha


preserve reentrant header used for actors ... through the service invocation process

In the dapr doc info, they show some reentrant actor scenarios. It looks like the dapr dotnet sdk automatically creates and passes the reentrant header id ... so as long as you follow the basic scenarios described in the docs it works fine.
ActorA -> ActorA
ActorA -> ActorB -> ActorA

https://docs.dapr.io/developing-applications/building-blocks/actors/actor-reentrancy/
I am wanting a do a slightly different scenario that includes a service invocation in the process.
ActorA -> invoke some service -> ActorA

This is currently not working because the reentrant header is not automatically preserved during a service invocation.

I actually have this working with out too many changes ... that said I am sure the implementation is not up proper standards ... is it worth submitting a PR or pointing to a fork to have folks take a look?

basically tuck away the reentrant header id here (before service invocation)
https://github.com/dapr/dotnet-sdk/blob/01cdc89710c16712442474afea97693f22c5852b/src/Dapr.Client/DaprClientGrpc.cs#L278
// aktxyz tuck away reentrant header id before service invocation
if (Dapr.Actors.ActorReentrancyContextAccessor.ReentrancyContext != null)
{
    request.Headers.TryAddWithoutValidation(Constants.ReentrancyRequestHeaderName, Dapr.Actors.ActorReentrancyContextAccessor.ReentrancyContext);
}

and then on the receiving end, pluck out the reentrant header id here
https://github.com/dapr/dotnet-sdk/blob/01cdc89710c16712442474afea97693f22c5852b/src/Dapr.AspNetCore/CloudEventsMiddleware.cs#L59
// aktxyz pluck out the reentrant header id after service service invocation
if (httpContext.Request.Headers.ContainsKey(Dapr.Client.Constants.ReentrancyRequestHeaderName))
{
    var daprReentrancyHeader = httpContext.Request.Headers[Dapr.Client.Constants.ReentrancyRequestHeaderName];
    Dapr.Actors.ActorReentrancyContextAccessor.ReentrancyContext = daprReentrancyHeader;
}

This seems to work fine ... so good enough to keep working ... would love to see support for this added in a proper way sometime in the future.

Does anyone on the DAPR team have any thoughts on this ... I think maintaining the reentrant header id across service invocations is a good thing !

I understand that this functionality is technical feasible. However, it would be really helpful if you could provide more details about the specific scenario where this shortcoming is blocking you and your project. This information would strengthen the case for addressing this issue, rather than it being seen as an interesting aspect of an otherwise incomplete but usable feature.

basically I have a case where I do something like this ...
ActorA -> invoke some service -> ActorA

ActorA receives a request
ActorA pubs out a message (service invoke), which is picked up by multiple services listening for this message type
when each service completes it's task, it sends results back to ActorA via a message

so basically
ActorA -> invoke some services (reentrant ID lost here) -> ActorA

Why do you need re-entrancy for this scenario? Why not simply call the actor anew with the message results?


Fork fraudulent NFT and scam notifier bot

https://app.forta.network/bot/0x112eaa6e9d705efb187be0073596e1d149a887a88660bd5491eece44742e738e
https://app.forta.network/bot/0x513ea736ece122e1859c1c5a895fb767a8a932b757441eff0cadefa6b8d180ac

scam notifier is forked
fraud nft trader to be forked tomorrow if we dont hear from david

Scam Notifier deployed: 0xdd83ac89fbc5dd028b7c4a711d4f0c5b13412b171801b126cb5f79b916920cd2
Manually added notifiers:

https://etherscan.io/address/0xba6e11347856c79797af6b2eac93a8145746b4f9
https://etherscan.io/address/0x666a3ce3f9438dccd4a885ba5b565f3035984793

Vasilis actively working on this

Fixed both bots.


[rom_ctrl,dv] Minor fixes for coverage improvement

1st commit sets max_outstanding_req of rom_ctrl register TLUL agent to 1. That is because TLUL agent for registers of rom_ctrl is autogenerated using tlul_adapter_reg. That module allows a max_outstanding_req of only 1.
2nd commit improves coverage score of onehot_fault_cg by setting option.at_least to 0 in coverpoints that we set option.weight 0.
Signed-off-by: Canberk Topal<EMAIL_ADDRESS>Needed to close up functional coverage of rom_ctrl

I added a second commit to this PR as it also improves coverage of rom_ctrl. It also improves coverage score of onehot_fault_cg by setting at_least option to 0 in coverpoints that we set weight 0. @weicaiyang I also added you as a reviewer as you might have a different idea about handling the second commit. Thanks for having a look in advance!

I added a second commit to this PR as it also improves coverage of rom_ctrl. It also improves coverage score of onehot_fault_cg by setting at_least option to 0 in coverpoints that we set weight 0. @weicaiyang I also added you as a reviewer as you might have a different idea about handling the second commit. Thanks for having a look in advance!

I haven't used at_least before. I guess this PR can bring the overall score to 100% if all coverpoint with weight > 0 are covered. Without this fix, as below, we can have instance coverage of this group to 100%, but score is less than 100%. Please correct me if I'm wrong.

Since you're improving coverage for rom_ctrl, can you take a look at this #16840 if you have a chance.

I added a second commit to this PR as it also improves coverage of rom_ctrl. It also improves coverage score of onehot_fault_cg by setting at_least option to 0 in coverpoints that we set weight 0. @weicaiyang I also added you as a reviewer as you might have a different idea about handling the second commit. Thanks for having a look in advance!

I haven't used at_least before. I guess this PR can bring the overall score to 100% if all coverpoint with weight > 0 are covered. Without this fix, as below, we can have instance coverage of this group to 100%, but score is less than 100%. Please correct me if I'm wrong.

Yes that's the case @weicaiyang thanks for having a look. I'll merge this PRas it got approved and move to work on the issue you pointed. Thanks everyone!

I added a second commit to this PR as it also improves coverage of rom_ctrl. It also improves coverage score of onehot_fault_cg by setting at_least option to 0 in coverpoints that we set weight 0. @weicaiyang I also added you as a reviewer as you might have a different idea about handling the second commit. Thanks for having a look in advance!

I haven't used at_least before. I guess this PR can bring the overall score to 100% if all coverpoint with weight > 0 are covered. Without this fix, as below, we can have instance coverage of this group to 100%, but score is less than 100%. Please correct me if I'm wrong.

Yes that's the case @weicaiyang thanks for having a look. I'll merge this PRas it got approved and move to work on the issue you pointed. Thanks everyone!

Great. That's quite annoying when reviewing coverage. Thanks for fixing that.


[SUPPORT] S3 slow file listing causes Hudi read performance.

Hudi MoR reading performance gets slower on tables with many (1000+) partitions stored in S3. When running simple spark.sql("select * from table_ro).count command, we observe in spark UI  that first 2.5 minutes no spark jobs gets scheduled and the load on cluster during that period is almost non existing.

When looking into logs to figure out what is going on during that period we observe that first  two and a half minutes Hudi is busy running HoodieParquetInputFormat.listStatus code link. I placed timer logs lines around various parts of that function and was able to narrow down to this line https://github.com/apache/hudi/blob/f5dc8ca733014d15a6d7966a5b6ae4308868adfa/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java#L103    this line execution takes over 2/3 of all time.
If I understand correctly what this line does it lists all files in a single partition.
Looks like this "overhead" is linearly depends on number of partitions as increasing number of partitions to 2000 almost doubles the overhead and cluster just runs HoodieParquetInputFormat.listStatus before starting executing any spark jobs.
To Reproduce
see code snippet bellow

Hudi version : master branch

Spark version : 2.4.4

Hive version : 2.3.6

Hadoop version : 2.8.5

Storage (HDFS/S3/GCS..) : S3

Running on Docker? (yes/no) : no

Additional context
    import org.apache.spark.sql.functions._
    import org.apache.hudi.hive.MultiPartKeysValueExtractor
    import org.apache.hudi.QuickstartUtils._
    import scala.collection.JavaConversions._
    import org.apache.spark.sql.SaveMode
    import org.apache.hudi.DataSourceReadOptions._
    import org.apache.hudi.DataSourceWriteOptions._
    import org.apache.hudi.DataSourceWriteOptions
    import org.apache.hudi.config.HoodieWriteConfig._
    import org.apache.hudi.config.HoodieWriteConfig
    import org.apache.hudi.keygen.ComplexKeyGenerator
    import org.apache.hadoop.hive.conf.HiveConf
    val hiveConf = new HiveConf()
    val hiveMetastoreURI = hiveConf.get("hive.metastore.uris").replaceAll("thrift://", "")
    val hiveServer2URI = hiveMetastoreURI.substring(0, hiveMetastoreURI.lastIndexOf(":"))
    var hudiOptions = Map[String,String](
      HoodieWriteConfig.TABLE_NAME → "testTable1",
      "hoodie.consistency.check.enabled"->"true",
      "hoodie.compact.inline.max.delta.commits"->"100",
      "hoodie.compact.inline"->"true",
      DataSourceWriteOptions.STORAGE_TYPE_OPT_KEY -> "MERGE_ON_READ",
      DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY -> "pk",
      DataSourceWriteOptions.KEYGENERATOR_CLASS_OPT_KEY -> classOf[ComplexKeyGenerator].getName,
      DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY ->"partition",
      DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY -> "sort_key",
      DataSourceWriteOptions.HIVE_SYNC_ENABLED_OPT_KEY → "true",
      DataSourceWriteOptions.HIVE_TABLE_OPT_KEY → "testTable1",
      DataSourceWriteOptions.HIVE_PARTITION_FIELDS_OPT_KEY → "partition",
      DataSourceWriteOptions.HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY → classOf[MultiPartKeysValueExtractor].getName,
      DataSourceWriteOptions.HIVE_URL_OPT_KEY ->s"jdbc:hive2://$hiveServer2URI:10000"
    )

    spark.sql("drop table if exists testTable1_ro")
    spark.sql("drop table if exists testTable1_rt")
    var seq = Seq((1, 2, 3))
    for (i<- 2 to 1000) {
      seq = seq :+ (i, i , 1)
    }
    var df = seq.toDF("pk", "partition", "sort_key")
    //create table
    df.write.format("org.apache.hudi").options(hudiOptions).mode(SaveMode.Append).save("s3://testBucket/test/hudi/zuyanton/1/testTable1")
    //update table couple times
    df.write.format("org.apache.hudi").options(hudiOptions).mode(SaveMode.Append).save("s3://testBucket/test/hudi/zuyanton/1/testTable1")
    df.write.format("org.apache.hudi").options(hudiOptions).mode(SaveMode.Append).save("s3://testBucket/test/hudi/zuyanton/1/testTable1")

    //read table
    spark.sql("select * from testTable_ro").count

@zuyanton this seems like a general issue with FileInputFormat
 int numThreads = job
        .getInt(
            org.apache.hadoop.mapreduce.lib.input.FileInputFormat.LIST_STATUS_NUM_THREADS,
            org.apache.hadoop.mapreduce.lib.input.FileInputFormat.DEFAULT_LIST_STATUS_NUM_THREADS);

can you try adding mapreduce.input.fileinputformat.list-status.num-threads=8 or something? (default inside Hadoop is 1)
cc @n3nash IIRC you mentioned a similar approach done at uber?

@zuyanton : HoodieParquetInputFormat relies on hadoop-mapreduce FileInputFormat listing implementation to perform listing. There is a knob in base FileInputFormat to tune listing parallelism.
"mapreduce.input.fileinputformat.list-status.num-threads"
The above config is set to 1 by default. Can you try increasing it to achieve speedup.
@zuyanton : We are also working on RFC-15 https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+and+Query+Planning+Improvements to holistically eliminate file listing and improve query performance.
cc @umehrot2  for any other suggestions.

@vinothchandar it didnt have any effect and I believe it shouldn't, since from what it looks like that parameter only gives improvement if you are trying to list statuses of multiple dirs https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L216 where is in our case its always one dir - the root location of single partition.

I think the finding by @zuyanton seems correct. Increasing the num-threads will not help because we just set the basepath of the table in the inputpath of jobconf. I believe we will have a good speed up, if instead of basePath we set all the partition paths in the inputpath of jobconf, and then increase the num-threads.
Another thing we can potentially explore is using Spark to perform this listing parallely on the cluster. But this seems like something we should target for 0.6.0 release with Blocker priority.

@zuyanton :  This sounds like a general  Spark/HMS query integration issue. Are we seeing similar behavior when running the same query over non-hudi table ?

@bvaradar we dont see similar issue with regular non hudi tables saved to s3 in parquet format. for regular tables "overhead" is the same and under one minute despite the number of partitions. Regular tables with 20k partitions as well as 100 partition take the same time to "load" before spark starts running its jobs where is hudi table on s3  becomes slow with 5k+ partitions. Although we use EMR 5.28 which comes with EMRFS s3 optimized committer enabled in spark by default ,so I assume whatever bottlenecks s3 has, are addressed in the committer.

Thanks @zuyanton  for the updates. IIUC, S3 optimized committer was for optimizing writes reducing the renames done. I might be wrong but I am generally curious on EMR optimizations for Spark. @umehrot2 : We can look at the option you mentioned regarding setting the partition paths and then increasing the num-threads. Is this one of the optimizations done internally within EMR spark ?

@zuyanton In your test you with regular parquet tables are probably not setting the following property in the spark config spark.sql.hive.convertMetastoreParquet=false. When you set this property to ```false`` only then will Spark use Parquet InputFormat as well as its listing code. Otherwise by default Spark uses its native listing (parallelized over the cluster) and parquet readers which are supposed to be faster.
However the way Hudi works is it uses InputFormat implementation. Thus for a fair comparison when you test regular parquet with Spark you should set spark.sql.hive.convertMetastoreParquet=false and I think you will observe quite similar behavior then as to what you are seeing. Would you mind trying that out once ?
But @bvaradar irrespective I think for Hudi we should always compare our performance against standard spark performance (native listing and reading) and not the performance of spark when it is made to go through InputFormat. So we need to get this fixed either ways if we have to be comparable to spark parquet performance which uses parallelized listing over the cluster.

@bvaradar @zuyanton EMR S3 optimized committer only helps avoid renames. Again that does not come into effect for Hudi because of the way Hudi datasource is implemented. Hudi datasource is not an extension of FileFormat datasource of Spark. It has its own commit mechanism and writing logic and does not use Sparks commit/write process. So EMR optimized committer unfortunately does not come into effect for Hudi workloads.
Irrespective the committer would not have any effect on this listing performance.

@umehrot2 you are right , with convertMetastoreParquet set to false , when querying regular parquet table with 20k partitions I can see similar behavior of spark not running any jobs for first 4 minutes.

@umehrot2 @bvaradar
Do you know if this problem will be solved in 0.7.0? I'm querying some big datasets with more than 500 partitions and I had the same problem.
Thank you

@umehrot2 @bvaradar
Do you know if this problem will be solved in 0.7.0? I'm querying some big datasets with more than 500 partitions and I had the same problem.
Thank you

@rubenssoto for some code paths, it will be. if you turn on hoodie.metadata.enable=true on the writing, you should see improvements. Hive queries should see improvement, SparkSQL with --conf spark.sql.hive.convertMetastoreParquet=false and --conf "spark.hadoop.hoodie.metadata.enable=true" should see improvement. Spark datasource path will see modest gains for now, integration coming quickly in 0.8.0. Will include it in release highlights

@rubenssoto for some code paths, it will be. if you turn on hoodie.metadata.enable=true on the writing, you should see improvements. Hive queries should see improvement, SparkSQL with --conf spark.sql.hive.convertMetastoreParquet=false and --conf "spark.hadoop.hoodie.metadata.enable=true" should see improvement. Spark datasource path will see modest gains for now, integration coming quickly in 0.8.0. Will include it in release highlights

@vinothchandar
Thank you so much for your answer.
When do you plan to release this version? I will try to make some workarounds until then.
Is this configuration right?
{ "conf": {
            "spark.jars.packages": "org.apache.spark:spark-avro_2.12:2.4.4",
            "spark.serializer": "org.apache.spark.serializer.KryoSerializer",
            "spark.jars": "s3://dl/lib/hudi-spark-bundle_2.12-0.8.0-SNAPSHOT.jar",
            "spark.sql.hive.convertMetastoreParquet": "false",
            "spark.hadoop.hoodie.metadata.enable": "true"}
}

I made these 2 queries:
spark.read.format('hudi').load('s3://ze-data-lake/temp/order_test').count()
%%sql
select count('*') from raw_courier_api.order_test
On the pyspark query spark creates a job with 143 tasks, after 10 seconds of listing the count was fast, but in the spark sql query spark creates a job with 2000 tasks and was very slow, is it a Hudi or spark issue?
Thank you so much!

@vinothchandar
Thank you so much for your answer.
When do you plan to release this version? I will try to make some workarounds until then.
Is this configuration right?
{ "conf": {
            "spark.jars.packages": "org.apache.spark:spark-avro_2.12:2.4.4",
            "spark.serializer": "org.apache.spark.serializer.KryoSerializer",
            "spark.jars": "s3://dl/lib/hudi-spark-bundle_2.12-0.8.0-SNAPSHOT.jar",
            "spark.sql.hive.convertMetastoreParquet": "false",
            "spark.hadoop.hoodie.metadata.enable": "true"}
}

I made these 2 queries:
spark.read.format('hudi').load('s3://ze-data-lake/temp/order_test').count()
%%sql
select count('*') from raw_courier_api.order_test
On the pyspark query spark creates a job with 143 tasks, after 10 seconds of listing the count was fast, but in the spark sql query spark creates a job with 2000 tasks and was very slow, is it a Hudi or spark issue?
Thank you so much!

0.7.0 is being voted on right now. Hopefully today.
So the spark.read.format('hudi') route (spark datasource path) does not go through Hive, so those configs may not help at all. Between pySpark and spark datasource in scala, there should be no difference. So not sure whats going on :/

0.7.0 is being voted on right now. Hopefully today.
So the spark.read.format('hudi') route (spark datasource path) does not go through Hive, so those configs may not help at all. Between pySpark and spark datasource in scala, there should be no difference. So not sure whats going on :/

0.7.0 is out!

0.7.0 is out!

With 0.7.0, one can set hoodie.metadata.enable to true to eliminate issues due to file listings. Closing this ticket now.


Fix test_perform_workload_action_read_only python test

Fails occasionally with IndexError: list index out of range
Drone build: https://drone-publish.rancher.io/rancher/rancher/10012/6/2

I Got another error on the same test.
>       raise ApiError(self._unmarshall(text))

E       rancher.ApiError: (ApiError(...), 'ServerError : Error updating workload under apps/v1 deployment:random-887325-1694029978:random-571539-1694029978 by rollback : Operation cannot be fulfilled on deployments.apps "random-571539-1694029978": the object has been modified; please apply your changes to the latest version and try again\n\t{\'baseType\': \'error\', \'code\': \'ServerError\', \'message\': \'Error updating workload under apps/v1 deployment:random-887325-1694029978:random-571539-1694029978 by rollback : Operation cannot be fulfilled on deployments.apps "random-571539-1694029978": the object has been modified; please apply your changes to the latest version and try again\', \'status\': 500, \'type\': \'error\'}')

https://drone-pr.rancher.io/rancher/rancher/32490/6/2

Another case in release/2.9

bad: https://drone-pr.rancher.io/rancher/rancher/37394/7/2
good: https://drone-pr.rancher.io/rancher/rancher/37396/1/1


Matplotlib and Plotly Charts

thank you @FeodorFitsner

You are welcome! The docs are coming later today.

Thanks @FeodorFitsner .... although interactivity is not possible ... still better to have something .... I hope some flutter chart library is possible in near future

Are the plans to support interactive charts (e.g. via WebView)?

This Package Supports Chart interactivity. https://pub.dev/packages/fl_chart
@FeodorFitsner mentioned that it could be implemented.

So, what do you guys mean by "interactivity"? There are multiple ways to interact with charts.

Examples of interactive charts:

https://plotly.com/python/plotly-express/
https://altair-viz.github.io/gallery/seattle_weather_interactive.html

As Altair charts can be saved as json/html ... I hope it can be rendered with vegaEmbed js library  or vega_embed_flutter

This Package Supports Chart interactivity. https://pub.dev/packages/fl_chart @FeodorFitsner mentioned that it could be implemented.

Interactive charts would be great!!


Release Notes Details

There are a lot more commits here than there should be, and it looks like we're mixing release and ILMT additions here as well.

Created by mistake. Please ignore. Closing the PR.


net/http: verify Transport discards HTTP/1.x connections upon request timeout

Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
go version go1.9.2 linux/amd64
Does this issue reproduce with the latest release?
no
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/git/go"
GORACE=""
GOROOT="installed/go"
GOTOOLDIR="/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build331671955=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
What did you do?
use   client = &http.Client{Transport: transport, Timeout: 60}  to invoke a http request
when timeout of this request has err info "Client.Timeout exceeded while awaiting headers" ,
it seems not  close the connection after timeout.
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.
client = &http.Client{Transport: transport, Timeout: 60}
resp, err := client.Do(req)
if nil != err {
logger.Errorf("traceId={%s},clientDoError={%s}", speech.TraceId, err.Error())
return respText
}
defer resp.Body.Close()
What did you expect to see?
when timeout occur  ,should close the connection,  not cache the timeout connection
What did you see instead?
the connection is not closed when timeout

I was able to reproduce that if a client request establishes a connection to a host, but the request times out, the connection is cached.
However, IMO I don't see why this behavior is undesirable or unexpected. If a connection is successfully established, I don't think it should be discarded just because a request happens to take a long time. If there's nothing wrong with the connection, why not keep it. If the first request that uses a connection times out, I agree there's a high likelihood that future requests will also timeout, but at least we would have the efficiency of the cached connection.
The code included by the OP is incomplete (@winworm , what is transport?). Below is code that is probably close to what was intended. Use -transpdialtimeout=30s -clienttimeout=1µs to simulate DefaultTransport and very short Client timeout.
var transpDialTimeout = flag.Duration("transpdialtimeout", 0,
	"transport dial timeout")
var clientTimeout = flag.Duration("clienttimeout", 0,
	"client timeout")

func main() {
	flag.Parse()
	// same as http.DefaultTransport except transpDialTimeout is configurable
	transp := &http.Transport{
		Proxy: http.ProxyFromEnvironment,
		DialContext: (&net.Dialer{
			Timeout:   *transpDialTimeout,
			KeepAlive: 30 * time.Second,
			DualStack: true,
		}).DialContext,
		MaxIdleConns:          100,
		IdleConnTimeout:       90 * time.Second,
		TLSHandshakeTimeout:   10 * time.Second,
		ExpectContinueTimeout: 1 * time.Second,
	}
	client := &http.Client{
		Transport: transp,
		Timeout:   *clientTimeout,
	}

	req, err := http.NewRequest(http.MethodGet, "http://example.com", nil)
	if err != nil {
		log.Fatal(err)
	}
	_, err = client.Do(req)
	fmt.Println(err)
        // include reading resp body and closing if err is not nil

	fmt.Println("Sleeping so you have time to look around at e.g. lsof -p PID ...")
	time.Sleep(2 * time.Minute)
}

thanks to meirf  ，Mainly because in some special cases, the client set the timeout specified, the client did not send the specified protocol data to the server (such as bandwidth limitations caused by the server did not receive), the server will continue to maintain the connection until the server set the timeout time (this time is greater than the client timeout setting). So we want the client to be able to control whether or not to close the connection over time and not into the pool.

I investigated this and wrote some tests (https://go-review.googlesource.com/c/go/+/123435) but was not able to reproduce. It seems to be doing the correct thing and closing the connection on timeout.
I'm going to close this.
Also, unrelated: the code &http.Client{Transport: transport, Timeout: 60} means 60 nanoseconds, not 60 seconds, which you probably discovered while reporting this bug I imagine.

Change https://golang.org/cl/123435 seems to broke all.bash occasionally both on amd64 and arm64.
Logs on amd64:
23:07:24 2018/07/17 22:07:21 http: panic serving <IP_ADDRESS>:55952: Fail in goroutine after TestClientTimeoutKillsConn_AfterHeaders has completed
23:07:24 goroutine 3032 [running]:
23:07:24 net/http.(*conn).serve.func1(0xc0004180a0)
23:07:24 	/home/root/ci-scripts/golang/src/net/http/server.go:1754 +0xd0
23:07:24 panic(0x859240, 0xc000eb4030)
23:07:24 	/home/root/ci-scripts/golang/src/runtime/panic.go:513 +0x1b9
23:07:24 testing.(*common).Fail(0xc00062ef00)
23:07:24 	/home/root/ci-scripts/golang/src/testing/testing.go:546 +0x135
23:07:24 testing.(*common).Errorf(0xc00062ef00, 0x90bcdf, 0x1e, 0xc0002e0cf0, 0x2, 0x2)
23:07:24 	/home/root/ci-scripts/golang/src/testing/testing.go:622 +0x83
23:07:24 net/http_test.TestClientTimeoutKillsConn_AfterHeaders.func1(0x9929c0, 0xc0009f02a0, 0xc000293500)
23:07:24 	/home/root/ci-scripts/golang/src/net/http/transport_test.go:4780 +0x338
23:07:24 net/http.HandlerFunc.ServeHTTP(0xc00031a060, 0x9929c0, 0xc0009f02a0, 0xc000293500)
23:07:24 	/home/root/ci-scripts/golang/src/net/http/server.go:1972 +0x44
23:07:24 net/http.serverHandler.ServeHTTP(0xc0005825b0, 0x9929c0, 0xc0009f02a0, 0xc000293500)
23:07:24 	/home/root/ci-scripts/golang/src/net/http/server.go:2740 +0xab
23:07:24 net/http.(*conn).serve(0xc0004180a0, 0x993200, 0xc0006b4300)
23:07:24 	/home/root/ci-scripts/golang/src/net/http/server.go:1855 +0x646
23:07:24 created by net/http.(*Server).Serve
23:07:24 	/home/root/ci-scripts/golang/src/net/http/server.go:2848 +0x2f5
23:07:24 FAIL
23:07:24 FAIL	net/http	3.217s
Logs on arm64:
13:10:14 --- FAIL: TestClientTimeoutKillsConn_AfterHeaders (0.06s)
13:10:14     transport_test.go:4793: Get error: Get http://<IP_ADDRESS>:35605: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
13:10:14 FAIL
13:10:14 FAIL	net/http	3.331s

@erifan if you haven't already, best to create a new issue instead of continuing the conversation here since the problems are different. This issue is about validating that Client.Timeout closes connections. The new separate issue would be about those validation tests being flaky.

@meirf OK, I'll do it in the future. I did not study this issue, but found that the test was cited here, so I thought it was the same question. Excuse me.


improve performance of conway's gol

can't do proper performance analysis since mac has awful tooling. Is blocked until I install linux VM

try using some mac tools

using samply found out that current implementation works for up to 250 ticks per second on 256x256 universe


vendor CompletableFutureUtil

This class seems to be removed from newer IntelliJ versions all of a sudden.
The runPluginVerifier task is failing because of it.

Anything ventured, we should also list in NOTICE.txt, no? I'm not entirely sure, but I think we should also include the original license notice in the vendored file.

Added the notice and the original license.


Size for pointer

Hey guys, updated atom to 1.1, and size for pointer changed:

Is it bug in atom or in some plugin?

Have you tried checking this in safe mode? (atom --safe) If you don't experience this in safe mode, then it's likely a package you have installed.

Is this using seti-syntax?

I have the same trouble. And yes, i'm using seti-syntax.

found the solustion:
https://github.com/jesseweed/seti-syntax/issues/106
@raphaelcuba

Thanks @pustovalov!


Syntax highlighting in notebook

I was able to get syntax highlighting in the console, but how do we enable it for the notebook?

This is done by CodeMirror -- neither Raku nor Perl 6 are in their list of languages, but it looks like some outside work has been done to support perl6

Ah, thanks.


Reword Python environment installation guide

The previous iteration tried to give instructions for a hermetic virtual environment used via ipykernel, but never gave instructions on how to do so.
This change removes the recommendation to register the virtual environment as a hermetic kernel, and instead recommends to install all necessary dependencies into the single environment, including Jupyter Notebook itself.
Some minor style changes were made to the rest of the setup instructions, including capitalization of Docker everywhere and fixing some incorrect grammar.

Still left TODO: Figure out how to correctly render the lists at the start of the notebook in mkdocs/mike.
Also I would like to discuss who is responsible for bootstrapping the requirements (esp. in CI) - do we substitute the install instruction with a !pip install --upgrade ... executable cell? This is cross platform AND works for both single and hermetic environments, but produces output which (as of now) is included in the documentation.

To finally address @Maciej818's comments (and to see why my responses are correct): The lakefs-spec version that is used in the notebook is the development version installed from the PR branch / commit, which carries all the latest changes - that is why the notebook is able to run to completion in CI.

Restructuring of the put events to be done in a follow-up.


Changes to copy

Old: Find health misinformation claims in YouTube videos.
New: Find health misinformation in YouTube videos
Old: Please enter a youtube url...
New: Enter a YouTube URL
Old: Get Claims
New: Analyse
Old: Previous Searches
New: History

Reopening this – let’s close it once it’s deployed.

Closing because this is deployed now.


Identify if we can reduce the number of contexts we create during model binding \ model validation

We create multiple contexts as we walk the object graph during binding and validation. These have shown up in perf in the past.
| 137 | 10237 | Reported By: Pranav K |

Duplicate of #1550 that has more data

@yishaigalatzer this isn't the same. #1550 is about the lifetime of a particular service. This bug was about the number of model binding contexts we new up in code (1-per-property) and the number of validation contexts we new up in code.


Add bill command to developer guide, Model diagram

Codecov Report
Base: 76.90% // Head: 76.90% // No change to project coverage :thumbsup:

Coverage data is based on head (1b051cb) compared to base (8b9ffda).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff            @@
##             master     #117   +/-   ##
=========================================
  Coverage     76.90%   76.90%
  Complexity      494      494
=========================================
  Files            76       76
  Lines          1433     1433
  Branches        152      152
=========================================
  Hits           1102     1102
  Misses          294      294
  Partials         37       37

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Issues in passing list(objects) variables via command option in Azure DevOps Pipelines

I identified that the issue is being caused by the Terraform command task in Azure Pipelines stripping the ‘double quotes’ in the command variables while passing it to the hosted Ubuntu agent.
#Variable:
variable "replication_locations" {
type = list(object({ location = string, failover_priority = number }))
default = []
}
Original Scenario”
INPUT Command Variable in the Pipeline task:
•	-var='replication_locations=[{ location = "ABC", failover_priority = 1}]'
Actual execution in the agent console:
•	-var='replication_locations=[{ location = ABC, failover_priority = 1}]'
So I tried escaping the double quotes using different ways with somethings like these:
•	-var='replication_locations=[{ location = "ABC", failover_priority = 1}]'
•	-var='replication_locations=[{ location = “"ABC"”, failover_priority = 1}]'
•	-var='”replication_locations=[{ location = \“ABC", failover_priority = 1}]”'
With the above  changes, I could VISUALLY see (as shown in green below) the variable getting passed correctly to the command line(with double quotes). However all of them are failing  as the terraform identifies/discovers some stray characters also as part of the input, though it’s not shown in the console.
Eg:
2019-10-05T11:03:42.6324045Z [command]/opt/hostedtoolcache/terraform/0.12.8/x64/terraform plan  -var='replication_locations_input=[{ location = "westus", failover_priority = 1}]'
2019-10-05T11:03:43.0815659Z [31m
2019-10-05T11:03:43.0816586Z [1m[31mError: [0m[0m[1mValue for undeclared variable[0m
2019-10-05T11:03:43.0817192Z [0mA variable named "'replication_locations_input" was assigned on the command
2019-10-05T11:03:43.0817378Z line, but the root module does not declare a variable of that name. To use
2019-10-05T11:03:43.0817621Z this value, add a "variable" block to the configuration.
2019-10-05T11:03:43.0818112Z [0m[0m
2019-10-05T11:03:43.0980533Z ##[error]Terraform command 'plan' failed with exit code '1'.:  Value for undeclared variable
2019-10-05T11:03:43.3250916Z ##[section]Finishing: terraform plan cosmos-acc-dbs
Tried the same process on both Windows and Ubuntu hosted agents n Az pipelines and getting the same problem.
Any idea how we can pass this object via command line variable in the extension?

Just came across this issue as well. In my case it's just a list of strings, but it removes the quotes and terraform doesn't understand the parameter.

We managed to use object notation in the task command option using the following syntax:
commandOptions: -var variable_name="{property1=\"$(PropertyA)\", propertyB=\"$(PropertyB)\"}"
 -var other_variable=$(otherValue)

Basically, you need to remove any quote/double quotes surrounding the whole commandOptions and you need to use " to surround the object and escape them for the properties inside the object.
Hope this helps!

I experienced the same problem trying to pass a list of client IP addresses to Terraform for configuration of firewall rules. The list is declared in a pipeline variable e.g.:
variables:
  - name: ClientIpAddressList
    value: '["<IP_ADDRESS>","<IP_ADDRESS>"]'

Passing the variable into the task as a Terraform env var provided a workaround, i.e.:
  - task: TerraformCLI@0
    displayName: Create Terraform plan
    inputs:
      command: plan
      workingDirectory: my-working-directory
      environmentServiceName: $(PipelineServiceConnection)
      commandOptions: >
        -out=tf.plan
    env:
      TF_VAR_client_ip_addresses: $(ClientIpAddressList)


fix null check in functions

What type of PR is this：

[x] bug
[ ] feature
[ ] enhancement
[ ] others

Which issues of this PR fixes ：
#4094 , #4092
Problem Summary(Required) ：
Some functions does not check null parameter correctly, cause BE crash.
The correct way to check the const value is FunctionContext::is_notnull_constant_column.

[FE PR Coverage check]
:heart_eyes: pass : 0 / 0 (0%)

[FE PR Coverage check]
:heart_eyes: pass : 0 / 0 (0%)


feat(ServiceDetail): Rename Instances to Tasks

Rename Instances to Tasks to avoid confusion and remove <FilterHeadline /> for services detail and pods detail.
Closes DCOS-17965
Before

After

Checklist

[x] Did you add a JIRA issue in a commit message or as part of the branch name?
[ ] Did you add new unit tests?
[ ] Did you add new integration tests?
[ ] If this is a regression, did you write a test to catch this in the future?

@leemunroe please take a look and let me know if anything else is needed. thanks.

Alfreddo test this please

Alfreddoson test this please

@bstavroulakis I fixed the failing tests. thanks.

@orlandohohmeier @Poltergeist @leemunroe putting hold merge on till somebody can explain the reasoning for removing the <FilterHeadline/> completely for service detail pages!

After a discussion I had with @orlandohohmeier offline, where we also had a into the screen designs, We realised that the filterHeadline should be removed with introducing the DSL filters to the task list. So for this PR we should keep the filter headline, and make sure the wording is Tasks.

Correct, the goal is to consolidate the filtering to the DSL filter component.
@ashenden has been doing some work on the design patterns for table headings/actions but I don't think we've reached a conclusion just yet. This would also affect any other collection tables e.g. nodes, components. For the purposes of this PR I would suggest keeping the headline and we make those changes separately.

@MatApple Let us know if you need more info. Thanks.

@weblancaster could you please adjust your PR to not remove the FilterHeadline as discussed in https://github.com/dcos/dcos-ui/pull/2441#issuecomment-335124670? We need apply these changes one after another and replacing the headline and filter buttons with the new filter component should be done in a separate PR.


Custom redirect URI scheme for mobile consumer not necessary for custom

Turns out, the redirect scheme for the mobile app can be https and is recommended in rfc8252: https://datatracker.ietf.org/doc/html/rfc8252#section-7.1
Basically, that custom scheme is only useful for private redirect Uris. It's not particularly helpful if you create an OAuth provider that allows for anyone to register an application because in that case you'll have to keep updating the redirect uri scheme with every new application (or override the django oauth toolkit Application to allow all redirect Uris which in this case isn't necessary).
When you take a look at section 7.2, it explicitly says:

App-claimed "https" scheme redirect URIs have some advantages
compared to other native app redirect options in that the identity of
the destination app is guaranteed to the authorization server by the
operating system.  For this reason, native apps SHOULD use them over
the other options where possible.

And in section 8.4:

For private-use URI scheme-based redirects, authorization servers
SHOULD enforce the requirement in Section 7.1 that clients use
schemes that are reverse domain name based.  At a minimum, any
private-use URI scheme that doesn't contain a period character (".")
SHOULD be rejected.

In addition to the collision-resistant properties, requiring a URI
scheme based on a domain name that is under the control of the app
can help to prove ownership in the event of a dispute where two apps
claim the same private-use URI scheme (where one app is acting
maliciously).  For example, if two apps claimed "com.example.app",
the owner of "example.com" could petition the app store operator to
remove the counterfeit app.  Such a petition is harder to prove if a
generic URI scheme was used.

Nevermind. I tried it; it did not work


don't wait for devices to settle when moving

for this application, when we call mv we should immediately continue rather than waiting for status objects to come back as done

abs_set stub


youtube-dl: 2020.11.24 -> 2020.11.26

Motivation for this change
https://github.com/ytdl-org/youtube-dl/releases/tag/2020.11.26
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[x] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 104931 1

3 packages failed to build:

bomi
python37Packages.tubeup
python38Packages.tubeup

35 packages built:

catt
celluloid
curseradio
haskellPackages.cut-the-crap
jellyfin-mpv-shim
jftui
kodi-cli
lollypop
minitube
mkchromecast
mopidy-youtube
mpc-qt
mps-youtube
mpv
mpvScripts.mpris
persepolis
plex-media-player
plex-mpv-shim
python27Packages.pafy
python27Packages.youtube-dl
python27Packages.youtube-dl-light
python37Packages.mpv
python37Packages.pafy
python37Packages.youtube-dl
python37Packages.youtube-dl-light
python38Packages.mpv
python38Packages.pafy
youtube-dl (python38Packages.youtube-dl)
youtube-dl-light (python38Packages.youtube-dl-light)
qimgv
somafm-cli
sublime-music
tartube
tuijam
ytcc

@AndersonTorres can you also take care of the backport? :)

@Ma27 looks fine but I never dit it before. Do you have a link to a tutorial or something?

@Ma27 looks fine but I never dit it before. Do you have a link to a tutorial or something?

https://github.com/NixOS/nixpkgs/blob/master/.github/CONTRIBUTING.md#backporting-changes

Result of nixpkgs-review pr 104931 run on x86_64-darwin 1

5 packages marked as broken and skipped:

haskellPackages.cut-the-crap
jellyfin-mpv-shim
kodi-cli
persepolis
sublime-music

4 packages failed to build:

mopidy-youtube
mpc-qt
python37Packages.tubeup
python38Packages.tubeup

19 packages built:

catt
curseradio
mkchromecast
mps-youtube
plex-mpv-shim
python27Packages.pafy
python27Packages.youtube-dl
python27Packages.youtube-dl-light
python37Packages.mpv
python37Packages.pafy
python37Packages.youtube-dl
python37Packages.youtube-dl-light
python38Packages.mpv
python38Packages.pafy
youtube-dl (python38Packages.youtube-dl)
youtube-dl-light (python38Packages.youtube-dl-light)
somafm-cli
tuijam
ytcc

Broken packages are either already fixed or require youtube-dlc

Was backported in 939603027de367119ad47adb3b5cda75d300c871.

Was backported in 939603027de367119ad47adb3b5cda75d300c871.


Enhancement: Non-normally distributed initial attitudes

Proposal
The only distribution currently allowed by core.NetworkModel.initialize_attitudes() is the normal distribution. There's no requirement that the attitudes be normally distributed, only symmetrically distributed, so support for additional distributions should be added.
Symmetric distributions to support

Uniform
Laplace
von Mises

Asymmetric distributions
Asymmetric distributions could be supported if attitude magnitude and attitude direction are sampled separately, with magnitude sampling from the asymmetric distribution and direction sampling from another distribution. This would enable supporting some nice distributions, e.g., log-normal, power-law, Weibull, but would require adding a new parameter for polarity sampling.

Resolved in PR #46


unresolved import module which i didn't promot in the file

no "fields" in this file at all. super weird

i totally have no idea about this. except this, all is well working.

This happens to me all the time. Usually the variable exists in another file and its like it somehow transfers over.


Seems like latest version or docs broken?

Why you close withour fixing it?

I raised a PR with the fix, the docs are automatically generated as part of the build. If you wait a few mins they will show up.

Working now.


TechZone Asset Creation & Approval--Asset Guild

Aligning to the Asset Guild process for TechZone ATK asset creation and deployment on TechZone.

Issues and request have been created in Asset Guild repository for 4 Automation (Reference Architecture, Integration, MAS and Data Fabric).  Reference issues (147 through 150) with delivery date of Nov 14 for draft release and Nov 30 for final release of these collections in Tech Zone.


Create Spring2023

+1 for creating a pull request. thanks

P.S. @AveryUALibrary  I added .md extension so it renders on github, here is the public link for sharing: https://github.com/ualibweb/UALIB_Workshops/blob/master/WorkshopSchedules/Spring2023.md


Member Filter

[Enter feedback here]
The document says v1.0/groups/{id}/members?$count=true&$filter=startswith(displayName, 'a') help to apply additional filter while querying however it throws erorr when i am trying to apply filter
"code": "Request_UnsupportedQuery",
"message": "The specified filter to the reference property query is currently not supported.",

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 51a32b12-951e-ff94-a7b8-c014968ce6cf
Version Independent ID: fea401c3-ce70-d7ce-4bae-c112308a711a
Content: List group members - Microsoft Graph v1.0
Content Source: api-reference/v1.0/api/group-list-members.md
Product: groups
Technology: microsoft-graph
GitHub Login: @Jordanndahl
Microsoft Alias: MSGraphDocsVteam

My bad !! ConsistencyLevel was not added. It would be better in document if this can be added


Fix/style issues

I fix the problem in the mobile view at the moment of executing the query in swagger. I have checked it with the developer tools but I couldn't check in localhost in my mobile since the styles are not rendering and I don't know why. Can you please check in yours before merging?

Hi @ekbalam11 ,
I think it looks nice:


TypeError: nativefier is not a function

Steps to reproduce:
~ npm install --save nativefier
Copying example of using nativefier as module. Executing:
nativefier(options, function(error, appPath) {
^
TypeError: nativefier is not a function
    at Object.<anonymous> (/Users/miguel/desarrollo/hubii-nativefier/index.js:25:1)
    at Module._compile (module.js:413:34)
    at Object.Module._extensions..js (module.js:422:10)
    at Module.load (module.js:357:32)
    at Function.Module._load (module.js:314:12)
    at Function.Module.runMain (module.js:447:10)
    at startup (node.js:139:18)
    at node.js:999:3

node -v -> 5.5.0
npm -v -> 3.6.0

I'm thinking if that has something to do with the fact that I've already nativefier installed as a global package...

node -v -> 5.5.0
npm -v -> 3.6.0

Sorry about the late reply, I've been quite busy with school nowadays. Could you try the following:
// note the `default`
var nativefier = require('nativefier').default;

nativefier({}, function(error) {
	console.log(error);
});

I believe the problem is due to the way I've exported the modules (written in ES6 and compiled to ES5 before being pushed to npm with Babel) which breaks compatibility with require. I need to read up more about this, could you try this workaround first?

Yeah, that's it. :+1:

This is related to https://phabricator.babeljs.io/T2212, I might add a babel plugin or something to fix the ugly .default syntax. Not sure what the rationale for why they killed it is though :/

Sadly, I'm not an expert with babel but you might add this plugin: https://www.npmjs.com/package/babel-plugin-add-module-exports

Yup! I'll just update the docs now to use the .default


Failed to find ROCm architecture from hipinfo / rocminfo. Defaulting to gfx1100.

Hello,
When running ROCm I'm getting this, multiple times:
Failed to find ROCm architecture from hipinfo / rocminfo. Defaulting to gfx1100.
Even though I have a working hipinfo, which on the bottom shows the architecture.

Code block collapsed, click to show

gcnArchName:                      gfx1100

Microsoft Windows [Version 10.0.19045.3570]
(c) Microsoft Corporation. All rights reserved.

C:\Users\Acrivec>hipinfo

--------------------------------------------------------------------------------
device#                           0
Name:                             AMD Radeon RX 7900 XTX
pciBusID:                         3
pciDeviceID:                      0
pciDomainID:                      0
multiProcessorCount:              48
maxThreadsPerMultiProcessor:      2048
isMultiGpuBoard:                  0
clockRate:                        2371 Mhz
memoryClockRate:                  1250 Mhz
memoryBusWidth:                   0
totalGlobalMem:                   23.98 GB
totalConstMem:<PHONE_NUMBER>
sharedMemPerBlock:                64.00 KB
canMapHostMemory:                 1
regsPerBlock:                     0
warpSize:                         32
l2CacheSize:                      4194304
computeMode:                      0
maxThreadsPerBlock:               1024
maxThreadsDim.x:                  1024
maxThreadsDim.y:                  1024
maxThreadsDim.z:                  1024
maxGridSize.x:<PHONE_NUMBER>
maxGridSize.y:                    65536
maxGridSize.z:                    65536
major:                            11
minor:                            0
concurrentKernels:                1
cooperativeLaunch:                0
cooperativeMultiDeviceLaunch:     0
isIntegrated:                     0
maxTexture1D:                     16384
maxTexture2D.width:               16384
maxTexture2D.height:              16384
maxTexture3D.width:               2048
maxTexture3D.height:              2048
maxTexture3D.depth:               2048
isLargeBar:                       0
asicRevision:                     0
maxSharedMemoryPerMultiProcessor: 64.00 KB
clockInstructionRate:             1000.00 Mhz
arch.hasGlobalInt32Atomics:       1
arch.hasGlobalFloatAtomicExch:    1
arch.hasSharedInt32Atomics:       1
arch.hasSharedFloatAtomicExch:    1
arch.hasFloatAtomicAdd:           1
arch.hasGlobalInt64Atomics:       1
arch.hasSharedInt64Atomics:       1
arch.hasDoubles:                  1
arch.hasWarpVote:                 1
arch.hasWarpBallot:               1
arch.hasWarpShuffle:              1
arch.hasFunnelShift:              0
arch.hasThreadFenceSystem:        1
arch.hasSyncThreadsExt:           0
arch.hasSurfaceFuncs:             0
arch.has3dGrid:                   1
arch.hasDynamicParallelism:       0
gcnArchName:                      gfx1100
peers:
non-peers:                        device#0

memInfo.total:                    23.98 GB
memInfo.free:                     23.86 GB (99%)

C:\Users\Acrivec>

I guess your code tries to call rocminfo, which is not included in windows package, thus erroring.
Or maybe hipinfo prints differently on linux?
For me the defaulting works, because it actually matches my architecture, but not for others.

Hi, Thanks for taking the time to report this issue.
#1950 should fix this.