Reading and writing to SQL database

We would like to read/write from an SQL database, we can rely on the https://github.com/MechanicalRabbit/FunSQL.jl package to help automate the writing of SQL code.

@TheCedarPrince since you are the local FunSQL expert, I wanted to ask, does FunSQL have a SQLNode object to do CREATE TABLE?  The feature that this issue is tracking is the very simple read/write functionality, essentially just read_json_acset and read_json_acset as exists in the current acset serialization functionality but for a general SQL DB connection. We'd need to use CREATE TABLE for each object in an acset.

One thing we will need to think about is how to handle writing of attribute data. SQLite, for example, has a rather restricted set of data types it supports (https://www.sqlite.org/datatype3.html). Unless we use BLOB, we are basically restricted to:
julia_to_sql_type(::Type{T}) where {T} = error("type $(T) not supported for SQL conversion")
julia_to_sql_type(::Type{T}) where {T<:Real} = "REAL"
julia_to_sql_type(::Type{T}) where {T<:AbstractString} = "TEXT"
julia_to_sql_type(::Type{T}) where {T<:Integer} = "INTEGER"

So any attributes that are unions, or vectors, etc, are out. One way to go about this is to let the user provide, if they want, for specific AttrTypes, an SQL target type and a conversion function (e.g. to convert a Vector{String} into a string delimited by commas. Otherwise we use the fallback above.

For acsets, we can more or less name the obs, homs, etc anything we want. But in SQLite (and other implementations), there are quite a few reserved words we cannot use for tables (see https://sqlite.org/lang_keywords.html). So by default we should name the table associated with an object the same name, unless the user gives an override.

Apparently SQLite does not allow multiple tables to be generated in a single statement, the code below (silently, of course) only generates the artist table and then ignores the remaining statements. This is a tricky problem for writing an arbitrary acset to a SQL database. The reason is that foreign keys play the role of Homs in acsets, and in SQL if you declare a column to be a foreign key to the primary key of another table, that other table had better exist already.
using SQLite

function get_table(db, table)
    query = DBInterface.execute(db, "SELECT * FROM $table")
    return DataFrames.DataFrame(query)
end

db = SQLite.DB()

DBInterface.execute(db, """
    CREATE TABLE artist(
        artistid    INTEGER PRIMARY KEY,
        artistname  TEXT
    );
    CREATE TABLE track(
        trackid     INTEGER,
        trackname   TEXT,
        trackartist INTEGER,
        FOREIGN KEY(trackartist) REFERENCES artist(artistid)
    );
""")

get_table(db, "artist")
get_table(db, "track")

One thing to do to get around this is to first create each table "individually" and later add the foreign key columns. Of course here there is another problem, in that FOREIGN KEY keyword does not seem to be compatible with ALTER TABLE. The best I can get working is below. This is less than ideal, as REFERENCES is a "column level" foreign key, meaning that the column in the target table it points to doesn't necessarily have to be that table's primary key. FOREIGN KEY is what declares that column to be pointing to another table's foreign key (this is not documented clearly anywhere, I read it at https://stackoverflow.com/questions/8595695/what-is-difference-between-foreign-key-and-reference-key).  So we still don't have proper foreign keys. @TheCedarPrince any thoughts?
using SQLite

function get_table(db, table)
    query = DBInterface.execute(db, "SELECT * FROM $table")
    return DataFrames.DataFrame(query)
end

db = SQLite.DB()

DBInterface.execute(db, """
    CREATE TABLE artist(
        artistid    INTEGER PRIMARY KEY,
        artistname  TEXT
    );
""")

DBInterface.execute(db, """
    CREATE TABLE track(
        trackid     INTEGER,
        trackname   TEXT
    );
""")

get_table(db, "artist")
get_table(db, "track")

DBInterface.execute(db, """
    ALTER TABLE track
        ADD COLUMN trackartist INTEGER DEFAULT NULL
        REFERENCES artist(artistid);
""")

Hey @slwu89 , here are some out of order thoughts on this!

@TheCedarPrince since you are the local FunSQL expert, I wanted to ask, does FunSQL have a SQLNode object to do CREATE TABLE?

FunSQL.jl does not have support for CREATE TABLE for I think precisely the reason we are running into -- all the variance between table creation in SQL flavors. If you want to get more clarification, the FunSQL.jl folks are very active on the Julia Zulip here. Otherwise, what they do support doing is that one can create a table first using whatever mechanism the SQL flavor we are using requires, and then we can insert the result of a FunSQL query into the table like this:
fun_sql = From(...) # Some FunSQL DSL statement

catalog = reflect(
    db_conn;
    schema = "",
    dialect=:sqlite
);

sql = render(catalog, fun_sql);

res = execute(db_conn,
    """
    INSERT INTO
        table
    SELECT
        *
    FROM
        ($sql) AS foo;
    """
)

But in SQLite (and other implementations), there are quite a few reserved words we cannot use for tables (see sqlite.org/lang_keywords.html). So by default we should name the table associated with an object the same name, unless the user gives an override.

Hm. We may need to create a small sanitizer package or routine that checks to make sure no statements are using reserved language per flavors we want to support.

One thing we will need to think about is how to handle writing of attribute data.

So one thing I am a strong advocate for here is to use DBInterface.jl interfaces to the DBs we are interested in using. I know that most DB packages within Julia support DBInterface interfaces which is great and should help a lot with this.

So we still don't have proper foreign keys. @TheCedarPrince any thoughts?

Is there any particular reason why we are sticking to SQLite? Could we perhaps use something like DuckDB which acts very similarly to PostgreSQL, supports in-memory databases, and should be fit for high performance use? I've encountered some of the roadblocks you were describing in SQLite and it is not a fault of FunSQL here but rather SQLite being strange in this regard.
This is just my thought on SQLite as we could switch to something else that "just does" what we want for a proof of concept. Or is SQLite on your side a design requirement?

Hey @TheCedarPrince, thanks!

Is there any particular reason why we are sticking to SQLite?

It's not a hard design requirement. I just started with it since of previous familiarity (though, obviously, not that familiar as I forgot about all these foot guns). It would be nice to support it nontheless given how widely it's used (and for https://github.com/AlgebraicJulia/AlgebraicRelations.jl/issues/33). But yes for a proof of concept it may be better to switch to something more "fully featured". I'll check out DuckDB!

Here is an example for creating a connection to DuckDB.jl:
using DBInterface
using DuckDB

db = DBInterface.connect(DuckDB.DB, ":memory:")

Let me know what else you need!

@TheCedarPrince testing out DuckDB a bit! Unfortunately it also has the Adding columns with constraints not yet supported notice. It's a problem because in general, there's nothing stopping someone from making an acset with a schema where each object has a hom to some other object, so there isn't a general way to "start with the tables that don't point to anything" and work backwards.
using DuckDB
using DataFrames
using Catlab

# create a new in-memory database
con = DBInterface.connect(DuckDB.DB, ":memory:")

julia_to_sql_type(::Type{T}) where {T} = error("type $(T) not supported for SQL conversion")
julia_to_sql_type(::Type{T}) where {T<:Real} = "REAL"
julia_to_sql_type(::Type{T}) where {T<:AbstractString} = "TEXT"
julia_to_sql_type(::Type{T}) where {T<:Symbol} = "TEXT"
julia_to_sql_type(::Type{T}) where {T<:Integer} = "INTEGER"

@present SchWeightedLabeledGraph <: SchLabeledGraph begin
    Weight::AttrType
    weight::Attr(E,Weight)
end

# to_graphviz(SchWeightedLabeledGraph)

@acset_type WeightedLabeledGraph(SchWeightedLabeledGraph, index=[:src, :tgt]) <: AbstractLabeledGraph

g = erdos_renyi(WeightedLabeledGraph{Symbol,Float64}, 10, 0.25)
g[:, :label] = Symbol.(collect('a':'z')[1:nv(g)])
g[:, :weight] = floor.(rand(ne(g)) .* 100)

# first make tables of each object and its attributes
for o in objects(acset_schema(g))
    o_attrs = attrs(acset_schema(g); from=o)
    stmt = Vector{String}()
    push!(stmt, "CREATE OR REPLACE TABLE $(o)(_id INTEGER PRIMARY KEY")
    for (col, _, type) in o_attrs
        push!(stmt, ", $(col) $(julia_to_sql_type(subpart_type(g, type)))")
    end
    push!(stmt, ");")

    DBInterface.execute(con, join(stmt))
end

# second add all homs (REFERENCE)
for (h, h_dom, h_codom) in homs(acset_schema(g))
    DBInterface.execute(con, """
        ALTER TABLE $(h_dom) ADD $(h) INTEGER REFERENCES $(h_codom)(_id);
    """)
end

Update:

Executing multiple statements in one execute or executemultiple call does not work in general regardless of database backend, see https://discourse.julialang.org/t/how-to-execute-sql-file/123675/, however the DuckDB-specific function DuckDB.query will do it
Interestingly, @TheCedarPrince discovered this several years ago https://github.com/JuliaDatabases/SQLite.jl/issues/303
It seems that neither SQLite or DuckDB currently supports using ALTER TABLE to add a foreign key. However, there may be work going on in DuckDB to allow this somewhat soon https://github.com/duckdb/duckdb/issues/57. Interestingly enough PostgreSQL does allow this, see their docs https://www.postgresql.org/docs/current/sql-altertable.html

Unsure where to go from here.


Windows certificate path contains extra characters

Description
The windows certificate installation contains an incorrect character (#460) and should be properly sanitized/trimmed.

Should be resolved in https://github.com/craftcms/nitro/commit/c2aaa4e5645e5bb59929d1302b9e8ecb13c54d67.

Just tagged 2.0.10 with this change!


Dispose should ref count rather than lock

Currently Dispose of OwnedMemory throws on Dispose if there are outstanding held references in Reserved Memory.
This should be internalised and all the Disposes though work independently; and the OwnedMemory should be disposed (or returned to pool) when the last one is called.
e.g. the originally issuer shouldn't have to wait for an upstream user to finish before discarding its own reference; its extra unnecessary logic.
Note: Not sure OnReferenceCountChanged is then needed in this scenario

@KrzysztofCwalina @davidfowl what is the current use case for the current dispose pattern? (i.e. I might be missing something)

Let's for a moment forget about names. We have two different operations we need to implement (either publicly or privately):

OperationA: shuts down the memory object (e.g. returns it to the pool, releases native memory, etc.)
OperationB: decrements the refcount.

OperationA needs to be at least protected virtual, so that subclasses (like the one storing GCHandles) can clean up additional resources. And so I claim that it is exactly what Dispose(bool) is and so it should be called DIspose(bool).
The semantics of dispose are prescribed in .NET. They are: calls Dispose(true). I think it's ok for dispose to do more than just call Dispose(true), e.g. zero-out some additional fields, but I don't think it is ok for it to not call Dispose(true).
And so OperationB cannt be Dispose(), as it does not always shut down the object.

Its not greatly safe though, currently you can go
var owner = new OwnedArray<byte>(1024);
Memory<byte> memory = owner.Memory;

var res0 = memorySlice.Reserve();
var res1 = memorySlice.Reserve();

var res2 = res0;
res0.Dispose();
res2.Dispose();

owner.Dispose(); // Disposes object, but res1 hasn't been disposed

Does https://github.com/dotnet/corefxlab/pull/944 resolved this satisfactorily?

@ahsonkhan, can you triage this issue? I would like to remove all System.Memory issues from corfxlab. thanks.

Closing this old issue now that Span/Memory is in corefx.
/cc @GrabYourPitchforks as FYI on discussion


Edit metadata to a project that is already in the DCP

Reasons to update metadata after submission, as a result of a human error. An example for these can be:
Incorrect cell count
Incorrect donor source for a sample
Acceptance Criteria
Provide a way to edit any metadata associated with a project or submission that is already in the DCP.
These metadata changes will be revalidated before they are transmitted to downstream services.
A version history is kept with the record of any edits and the time they were made.
Existing UUIDs remain the same

Story sliced the wrong way. Closing, and will re-think this.


xt/aspell.t doc/Language/unicode_entry.pod6

The problem
$ xt/aspell.t doc/Language/unicode_entry.pod6

Fails after b4534dbe. Looks like commands that need to be added to the dictionary file.

You mean the rxvt configuration commands?

I didn't read the file or scope the words, just saw the failing test.

OTOH, is it worth the while to spell-check code blocks?

IMO yes, because there's a lot of items in code that are words, like comments, variable names, etc.
Even if they're not words, having the config "words" whitelisted means there's less chance of someone typo-ing them in other sections of the document.

Shouldn't we maybe have some special :lang keyword that just skips the
whole block? In cases such as this, it's really cumbersome to add every one
of them (or even whitelist them).

I'm not sure rewriting the test to avoid code helps, given the large amount of code we do want tested. (Including use of official Perl 6 class names in code, e.g.). I don't think that the overhead on keeping the spelling test passing is that great, even in this particular case.
The only reason I posted this as a ticket was so that the original author could verify the spelling of the comands/params before we codified it.

Looks like I missed a couple of colons in the XTerm config, but other than that there are no typos. My bad

Looks like @Kaiepi fixed his own typo note above in<PHONE_NUMBER>f304f8a23a54bdcb443b6a885fb966


Manual Python 3.9 Migration

Checklist

[ ] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

@conda-forge-admin, please rerender

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.
I do have some suggestions for making it better though...
For recipe:

License is not an SPDX identifier (or a custom LicenseRef) nor an SPDX license expression.

Documentation on acceptable licenses can be found here.

@conda-forge-admin restart ci

@conda-forge-admin restart ci

The atomman code is not yet Python 3.9 ready.
2020-11-02T10:54:08.6261466Z     atomman/core/dmag.c: In function '__Pyx_modinit_type_init_code':
2020-11-02T10:54:08.6263339Z     atomman/core/dmag.c:17305:26: error: 'PyTypeObject {aka struct _typeobject}' has no member named 'tp_print'; did you mean 'tp_dict'?
2020-11-02T10:54:08.6264575Z        __pyx_type___pyx_array.tp_print = 0;
2020-11-02T10:54:08.6265153Z                               ^~~~~~~~
2020-11-02T10:54:08.6265805Z                               tp_dict
2020-11-02T10:54:08.6280221Z     atomman/core/dmag.c:17310:32: error: 'PyTypeObject {aka struct _typeobject}' has no member named 'tp_print'; did you mean 'tp_dict'?
2020-11-02T10:54:08.6281507Z        __pyx_type___pyx_MemviewEnum.tp_print = 0;
2020-11-02T10:54:08.6281873Z                                     ^~~~~~~~
2020-11-02T10:54:08.6282265Z                                     tp_dict
2020-11-02T10:54:08.6292893Z     atomman/core/dmag.c:17325:31: error: 'PyTypeObject {aka struct _typeobject}' has no member named 'tp_print'; did you mean 'tp_dict'?
2020-11-02T10:54:08.6293746Z        __pyx_type___pyx_memoryview.tp_print = 0;
2020-11-02T10:54:08.6294108Z                                    ^~~~~~~~
2020-11-02T10:54:08.6294440Z                                    tp_dict
2020-11-02T10:54:08.6312904Z     atomman/core/dmag.c:17338:36: error: 'PyTypeObject {aka struct _typeobject}' has no member named 'tp_print'; did you mean 'tp_dict'?
2020-11-02T10:54:08.6313544Z        __pyx_type___pyx_memoryviewslice.tp_print = 0;
2020-11-02T10:54:08.6314020Z                                         ^~~~~~~~
2020-11-02T10:54:08.6314371Z                                         tp_dict


Add some packages from cardano-wallet to CHaP

This will fix https://github.com/cardano-foundation/cardano-wallet/issues/4411

Hi @abailly, thank you for this PR! I am wondering about it's status, is there something pending or are we awaiting approval?

AFAIK we are waiting approval, but this is beyond my karma level :)

cc @input-output-hk/cardano-haskell-packages-trustees
Might be worth creating a CF team & giving update privileges to it for packages added by this PR.

Looks like you need a rebase


Bug fix java-udf memory leak

Proposed changes
have new object twice, but only have one delete.
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


Regen

Description
Regenerated loot table files after forge version change removed: "functions": [] from each file.
Checklist:

[ ] My code follows the style guidelines of this project (.editorconfig, most IDEs will use this for you)
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation

Thanks :)


Update CreateCommand.java

Fixed incorrect comparison of NAME_MAX_LENGTH and actual name length that lead to players being unable to create factions with shorter names than the cap.

apologies for taking so long, I'll merge this now


Update for ipywidgets 8

ipywidgets 8 is the default now. The old widgets fail to run because they're out of date. They just silently produce no output in the notebook, while printing an error to the browser's console about version mismatches.
They run with these changes. I tested every widget in the example notebooks.
The basic messaging hasn't changed. Some minimum versions are different and a few fields have changed at the root of the tree. We've moved from https://github.com/jupyter-widgets/ipywidgets/blob/main/packages/schema/jupyterwidgetmodels.v7-5.md to https://github.com/jupyter-widgets/ipywidgets/blob/main/packages/schema/jupyterwidgetmodels.v8.md
The big changes have to do with tabbable, tooltips, and html descriptions being made available in most widgets. Borders was split into four, one for each direction.
NB: I did not update the example notebooks to reflect this change.

This looks great, thanks for doing this! Let me figure out what's happening with those failing builds (seems like a system dependency issue unrelated to this change) and I am happy to merge once that is resolved.

Thank you!!


Stop silently swallowing exceptions when looking up schemas

This line

It looks like the point of swallowing the exception was to catch the FileNotFoundExceptions which occur when the schema isn't in the repo. The handling should be restricted to FileNotFoundExceptions rather than all Throwables.


API Endpoints for collaborators

Adds the following endpoints:

GET repos/:username/:reponame/collaborators: list collaborators
GET repos/:username/:reponame/collaborators/:collaborator: check if a user is a collaborator
DELETE repos/:username/:reponame/collaborators/:collaborator: remove a collaborator

These endpoints are part of the Github API

Relevant functions needed in go-sdk as well :)

But there is no permission check?

@lunny Fixed. I also added a check for the existing PUT repos/:username/:reponame/collaborators/:collaborator endpoint.

LGTM

LGTM


Upgrading to Terraform 0.12: separate configuration file for Terragrunt?

Hi! I'm one of the engineers at HashiCorp who works on Terraform Core.
As you might be aware, we've been working for some time now on various improvements to the Terraform configuration language, to address a number of usability problems that have been reported over the years. The new changes are broadly compatible with existing configurations, and particularly those using the common idioms reflected in the Terraform documentation.
Unfortunately resolving some of these problems required tightening up some things that were previously rather loose. Current and previous versions of Terraform ignore certain things that are considered invalid, often producing surprising results, and so to improve usability the new implementation has some more error checking in order to catch mistakes and produce error messages that are hopefully more actionable.
We're planning to include a migration helper tool in the next release in order to help detect and fix some of these rough edges. The details of what this tool does will be largely informed by real-world usage, and so we're planning to release an early preview of the new implementation as soon as it is in a usable state in the hope that we can gather as many examples as possible of patterns in the wild that the migration helper tool should be looking for.
With all of that context in mind, we know that there are lots of users wrapping Terraform with Terragrunt, and so we want to make sure there's a good upgrade path. I'm opening this issue to discuss a specific problem I've noticed, but I'd like to discuss any other potential hurdles too, in whatever forum makes sense.

Terragrunt currently uses an extension of the terraform.tfvars file as its own configuration file, relying on the fact that Terraform itself ignores definitions in that file that are not defined as variables, and then using a special variable name terragrunt whose value is not, by Terraform's rules, valid: it includes mixed-typed nested structures, has nested blocks, etc.
The new implementation has a few differences that will cause friction here if we proceed naively:

Terraform will consider the mixed-type nested block structures as invalid and will generate error messages about them during validation, since they are invalid by Terraform's rules.
The new configuration parser distinguishes more strongly between object expressions and nested blocks, in order to resolve various confusing behaviors that resulted from the ambiguity of that in the original HCL design.
The expression evaluator and the structural language were previously entirely separate (HIL vs. HCL), but they are combined in the new implementation, which means that interpolation sequences ${ are interpreted in all contexts, with the difference now being whether variables and functions are available, rather than whether interpolations are possible at all.

Looking at the "live infrastructure" example, a naive syntax-level pass of migration would produce the following rewritten version:
# Terragrunt is a thin wrapper for Terraform that provides extra tools for working with multiple Terraform modules,
# remote state, and locking: https://github.com/gruntwork-io/terragrunt
terragrunt = {
  # Configure Terragrunt to automatically store tfstate files in an S3 bucket
  remote_state = {
    backend = "s3"
    config = {
      encrypt        = true
      bucket         = "terragrunt-example-prod-terraform-state"
      key            = "$${path_relative_to_include()}/terraform.tfstate"
      region         = "us-west-2"
      dynamodb_table = "terraform-locks"
    }
  }
  # Configure root level variables that all resources can inherit
  terraform = {
    extra_arguments = {
      # (this part will need to do some heuristic processing to decide whether to use
      # direct objects or lists of objects, mimicking the behavior of the current hcl.Decode
      # into an interface{}, so the details may not exactly match this in practice.)
      bucket = {
        commands = ["$${get_terraform_commands_that_need_vars()}"]
        optional_var_files = [
          "$${get_tfvars_dir()}/$${find_in_parent_folders("account.tfvars", "ignore")}"
        ]
      }
    }
  }
}

Since Terragrunt is using the HCL parser directly to process its configuration file today, it has the freedom to interpret the tree of HCL AST objects inside here as desired, and so I expect there are expectations here that are different than how Terraform generally deals with HCL structures it has no specific schema for. Also, due to the fact that interpolation syntax is now accepted in all quoted strings (variables and functions are just not available in some contexts), the naive migration tool will escape the ${ sequences under the assumption that the user intended them to be treated literally, since that was Terraform's own interpretation of a .tfvars file before this change.
Therefore it feels to me like the most practical path to take here is for our migration tool to have a special case for a variable called terragrunt (possibly activated only if there is no variable "terragrunt" block in configuration) which can then have some light understanding of Terragrunt's configuration structure and make some allowances for it so that we can avoid unintentional breakage of the Terragrunt configuration.
Terragrunt is currently relying on syntax features that are considered invalid in the new implementation, so leaving the configuration untouched by the migration tool would inevitably lead to syntax errors during parsing.

In order to avoid conflicts between Terraform's and Terragrunt's interpretation of the same file, I'd like to propose having Terragrunt look for a Terragrunt-specific configuration filename (e.g. terragrunt.conf) in preference to terraform.tfvars, and then Terraform would entirely ignore this file and leave Terragrunt's own parser to deal with it. Terragrunt could continue to use the original implementation of HCL, avoiding the need to deal with any of the new processing rules from the new implementation.
If we were to go down this path, we could build the configuration migration tool so that its special case for the terragrunt variable name is to migrate the contents out into a terragrunt.conf file that retains the block structure and original-HCL syntax that Terragrunt is expecting, and then deletes this value from the terraform.tfvars file altogether so that Terraform itself won't try to parse it.
I understand that this is a pretty drastic shift in the design and usage of Terragrunt, so I wanted to start talking about this early so we have some time to think through the implications and any other alternative approaches. I hope that together we can find a reasonable path forward!

Hi @apparentlymart! First, a big thank you for checking in to make sure there's a migration path for Terragrunt! 🍺
Second, could you provide more details on what HCL and Terraform changes are coming? Terragrunt exists as a quick way to fill in gaps in Terraform, but as those gaps get filled in, we're only too happy to reduce the need for Terragrunt, and hopefully have it go away entirely in the future.
Once that's clear, we can think about whether changing the config format or moving to a different config file makes more sense.

However, I can say that the focus of this release is on configuration changes rather than workflow changes, and so I believe all of Terragrunt's use-cases continue to make sense for the moment. Once the configuration language is in good shape we are planning to divert our attention to workflow for a while, at which point we'll definitely be interested in discussing if some or all of Terragrunt's features make sense for incorporation into Terraform. There'll be at least one more major release cycle before any changes that might reduce the scope of Terragrunt, and so I think it's worthwhile to ensure there's a viable path forward .

Understood. Thx for providing the context.

My main motivation with this proposal was to try to minimize the need for tight co-ordination between the Terraform release and any Terragrunt changes, and that was why I lent towards a separate file as a way to leave Terragrunt in control of its own destiny with regards to its configuration parsing, rather than having a single file that needs to be understandable both to Terraform's new parser (based on HCL2) and Terragrunt's existing parser (based on the original HCL implementation).

The main concern I have with a separate file is that it will double the number of files necessary to use Terragrunt. One of the main complaints we have is that there are already too many files/folders to manage, so I'm a bit hesitant to make it worse.
I'm going to take a minute to think out loud and throw out some ideas:

One of the primary use cases for Terragrunt "remote Terraform configurations", where you can have a single, versioned definition of your Terraform module in one place and deploy that module across environments by defining a separate terraform.tfvars file for each environment (see, Keep your Terraform code DRY). With this use case, Terragrunt checks out your Terraform code into a temp folder, copies the terraform.tfvars file into that temp folder, and then runs terraform. With this use case, it would be easy for Terragrunt to remove the terragrunt-specific code from terraform.tfvars before running terraform. This allows us to do whatever we want in terraform.tfvars and decouples the release.

The approach above doesn't solve the use case for people who don't use the remote Terraform configurations feature. We could update Terragrunt to always copy code to a temp folder before running; might be a bit clunky and have side effects.

If the HCL2 parser supports parsing of interpolation, that would actually be a great help, as currently, Terragrunt uses the HCL parser to read the general structure of the .tfvars file, but processes interpolations through hacky, sometimes buggy regular expressions 😁

On the other hand, while the structural changes to the Terragrunt configuration you posted (mainly adding = in a few places?) all seem fairly minor, the double dollar sign for interpolation is a bit annoying and would not allow Terragrunt to use HCL2 to parse the interpolations.

On the topic of interpolations, what will be allowed in terraform.tfvars?

The main concern I have with a separate file is that it will double the number of files necessary to use Terragrunt. One of the main complaints we have is that there are already too many files/folders to manage, so I'm a bit hesitant to make it worse.

fwiw, I know I'd have no problem using a separate config file. I rather like the clarity of separating the two more cleanly. Personally, I don't use it myself to define variables anyway, since I like maintaining the separation of concerns. Instead, I use a separate file <foo>.auto.tfvars, which terraform reads automatically...
And technically, a separate config file wouldn't necessarily double the number of files, since terraform.tfvars would no longer be required...
How about a poll, maybe just using reactions on the issue, or perhaps publishing a poll through the newsletter?

And technically, a separate config file wouldn't necessarily double the number of files, since terraform.tfvars would no longer be required...

How so? You still need to configure Terragrunt, so that's one file. I suppose if your module needed no variables, then there wouldn't be a second, but that doesn't seem like a common use case.

Pass the vars inside the terragrunt config, using extra_arguments. :)

Ah, I gotcha. That's something worth considering, I suppose. Terragrunt as it's own, completely separate format for (a) downloading Terraform configurations from somewhere, (b) configuring remote state for them, and (c) running apply with a bunch of arguments passed using -var...

I've been thinking about this a bit more and moving to a separate file, along with some changes to Terragrunts "ergonomics", may make sense.
Here's a rough idea of what a typical "child" terragrunt.hcl may look like:
# Include all the settings from the parent config
include = "${find_in_parent_folders()}"

# The Terraform code to use
source = "git::git@github.com:foo/modules.git//app?ref=v0.0.3"

# Variables to pass to the Terraform code
variables = {
  instance_type = "t2.micro"
  instance_count = 5

  # A new helper to get output variables from other Terragrunt / Terraform modules
  vpc_id = "${get_output("../vpc", "vpc_id")}"

  # A new helper to get input variables from other Terragrunt HCL files
  account_id = "${get_variable("../account.hcl", "account_id")}"
}

# Keep your CLI flags DRY
extra_arguments = {
  retry_lock = {
    commands  = ["${get_terraform_commands_that_need_locking()}"]
    arguments = ["-lock-timeout=20m"]
  }
}

The main changes are:

The config lives in a separate .hcl file. The default name should be terragrunt.hcl.
Most of the config settings are now flattened, top-level items to avoid unnecessary nesting.
The variables that used to be listed directly in terraform.tfvars now go in a variables = { ... } block.
There are new helpers to read variables from other Terragrunt .hcl files, which have their own variables = { ... } blocks, and new helpers to read outputs, by running terraform output or terragrunt output, from other Terraform/Terragrunt modules.

Note that this requires code changes in Terragrunt itself. We don't need all of these for it to be useful, but at the very least, what file it reads, and the structure of that file would need to be available immediately. The new helpers could come later.
I'd love to hear feedback from any Terragrunt users.

@apparentlymart What is the rough timeline for HCL2?

Seems reasonable to me! I suppose the new "hooks" feature would be specified something like extra_arguments? But as a list of dictionaries, or an ordered dictionary, since execution order might matter... Not sure I have the syntax right for that in HCL2, but maybe:
after_hooks = [
  "foo" = {
    commands = [...]
    execute = [...]
  }

  "bar" = {
    commands = [...]
    execute = [...]
  ]

before_hooks = [
  "abc" = {
    commands = [...]
    execute = [...]
  }

  "bcd" = {
    commands = [...]
    execute = [...]
  }
]

Ah, right, I left out hooks, and yes, they would look exactly like that.

That looks great to me, @brikis98! The HCL2 parser should be able to accept that format, though I'd make some small adjustments to some details for "idiomatic" HCL2:
# No need to wrap a single expression in interpolation
include = find_in_parent_folders()

source = "git::git@github.com:foo/modules.git//app?ref=v0.0.3"

# Fixed elements (part of the structure of the format) tend to be defined as blocks rather than
# attributes, so no = sign here.
inputs {
  instance_type  = "t2.micro"
  instance_count = 5
  vpc_id         = get_output("../vpc", "vpc_id")
  account_id     = get_input("../account.hcl", "account_id")
}

extra_arguments {
  retry_lock {
    commands  = get_terraform_commands_that_need_locking()
    arguments = ["-lock-timeout=20m"]
  }
}

remote_state {
  backend = "s3"
  config {
    bucket = "my-terraform-bucket"
    region = "us-east-1"
    key    = "${path_relative_to_include()}/terraform.tfstate"
  }
}

# Not exactly sure about these, but their structure suggests something I'd model
# with repeated nested blocks rather than single attributes.
hook_after "foo" {
  commands = [ ... ]
  execute  = [ ... ]
}
hook_before "abc" {
  commands = [ ... ]
  execute  = [ ... ]
}

We're not promising compatibility for the HCL2 package APIs until some time after Terraform is released using it, in case we find design flaws during implementation in Terraform. With that said, we've already completed most of the work in Terraform that interacts with the HCL2 parsing APIs, and so in practice significant changes from here on out are not very likely.
The structure here seems rigid enough that it could probably be processed using the higher-level gohcl package, which uses Go's reflect package to drive decoding via structs. (Terraform's own processing is a little more involved and so it goes to the lower-level API so it can evaluate things gradually during the graph walk.) Since Terraform's needs are the main thing driving HCL2 development right now the gohcl package is not as well documented as it ought to be, but I think its complete enough to be able to process the format you described here, using DecodeBody with a set of structs that represent the intended structure, using functions defined in the EvalContext. Either way, the first step is to parse a file to get a "body" using the parser API.
We are currently in the process of updating Terraform Core to support HCL2's new type system, and so getting to the other end of that is the main blocker for us releasing the initial preview release, and there's still quite a lot of work to do here. How long it'll be until the final release after that depends on what sort of feedback we get from that preview release, but making time for the usual beta and release candidate process we do for any major release means that it'll be another couple months at least before we'll be final here. (The work is driving the schedule rather than the other way around, so unfortunately I can't be more precise at this stage.)

Thanks for the feedback @apparentlymart. The new syntax looks really nice, especially for simple expressions/interpolations 👍
I think I'm on board with moving to this format. We're a bit overloaded now, so probably won't be able to dig into this for at least a few more weeks, but it sounds like Terraform moving to HCL2 is still a little ways down the road, so I think we have time.
Two more questions for you:

You mentioned some sort of migration tool. Are you building something to parse terraform.tfvars and transform it accordingly? If so, it would be very helpful if there was a way to use that tool to detect terraform.tfvars files with a terragrunt = { ... } block and transform them to the new terraform.hcl format automatically.

Do the HCL2 libraries support parsing only, or also modifying the AST in memory, and writing it back out? The original HCL library could do pretty-printing of the AST, but not if you made any modifications to it.

The planned migration tool is primarily for updating .tf files, but we were planning to also have it do some simple processing on .tfvars files just to deal with the fact that the HCL2 parser will now interpret ${ sequences in there. We had honestly not planned to do any super-complex processing of .tfvars files, but if there's a reasonably straightforward and well-defined way to map a terragrunt variable assignment onto a terraform.hcl file I'm certainly game to try it in order to smooth the transition here.
The original HCL parser did, as you noted, have a few issues with the generation portion. For HCL2, given the increased complexity of handling both structural and expression parsing in both HCL native and JSON syntaxes, we decided not to build writing support into the main API. Instead, we have a separate package hclwrite which offers an API more tailored for generation and for making surgical changes to existing files. However, this package is not yet fully-functional since again Terraform's needs are driving the priorities for now and the terraform fmt needs can be mostly met by the relatively-simpler Format function. The hclwrite package design and functionality will be finished up in due course, probably driven by some configuration-generation use-cases like hashicorp/terraform#15608.
For the sake of the migration helper tool, we are planning to generate HCL2 syntax output in a more template-ish way, rather than generating it with the hclwrite package, just because the needs there are pretty constrained and it reduces the size of the work required for what will hopefully be a pretty short-lived codepath. Our prototypes so far produced roughly-formatted HCL syntax using templates and then ran the result through Format to get back to idiomatic style.

A question on the inputs block--will all inputs blocks be be merged from all included terragrunt.hcl files?  Seems like that would make it so you could reference those values similarly to terraform's locals, like ${input.myvalue} rather than having to specify what file it comes from with ${get_input("my_terragrunt_source", "myvalue")}.  Or was get_input intended to access more than just values in the inputs block?

A question on the inputs block--will all inputs blocks be be merged from all included terragrunt.hcl files?

Good question. Currently, Terragrunt only allows you to include a single file. I suppose inputs would be merged from that one file.

Seems like that would make it so you could reference those values similarly to terraform's locals, like ${input.myvalue}

An interesting idea. I hadn't thought through what the syntax would be likely to reference inputs defined in the current file or imported from the parent. We could use get_input("myvalue") to indicate the current file or input.myvalue.

Oh I was under the impression you could have more than a single layer of includes.  Is that not what this test does?
https://github.com/gruntwork-io/terragrunt/tree/master/test/fixture-parent-folders/multiple-terragrunt-in-parents

just as @lorengordon explained, i also separate the concerns by making sure any variable required by my modules are placed in separate files pulled by terragrunt using extra_arguments as in
    extra_arguments "conditional_vars" {
      commands = ["${get_terraform_commands_that_need_vars()}"]

      required_var_files = [
        "${get_tfvars_dir()}/${find_in_parent_folders("global.tfvars")}",
        "${get_tfvars_dir()}/${find_in_parent_folders("env.tfvars")}"
      ]

      optional_var_files = [
        "${get_tfvars_dir()}/component.tfvars"
      ]
    }

Having all the variables in a terragrunt/terraform-logic-free separate file enables a clear separation between the modules code and the variables and eases their use by ops users on a day-to-day basis (read: I build modules and set up overall solutions <--> end-users fine-tunes the variables)
So i won't use the new input block much.
Reading the lines in this thread, i expect this won't be impacted much by the proposed changes so I have no concerns to use a separate terragrunt config file.

Now, regarding the new helper get_output(), I wonder: could we make this interpolation function work inside the .tfvars files used to store the modules variables only? I guess the separation of concerns postulate would dictate along these lines not to interfere with terraform files anymore, then it would be more a question of an additional feature to add to terraform's core, and the proposed terragrunt implementation should be seen as an interim solution until then.
Thoughts?

@ebarault I'd love to see some interpolation in .tfvars files. Looking up ENV values, and many of the "data" resources would be killer. I'd especially like to be able to retrieve values from the SSM Parameter Store. It's quite a pain to expose these implementation-specific "value lookup" constructs in a module (especially if you publish it publicly). Getting this in terraform core would be awesome! 👍 👍 👍

as of now, i use terragrunt hooks to interpolate "remote" vars in .tfvars files: more specifically all vars nested in more or less complex data structures than cannot be easily used with remote data sources and current terraform's HCL implementation

No progress yet.

Terraform 0.12 will go through an alpha release, at least one beta release, and at least one release candidate before final, which'll add up to at least a month and a half of release process, so please don't worry about it just surprisingly appearing one day!
The terragrunt-specific configuration migration mechanism we discussed in this thread is unlikely to make it in before the alpha, because the focus of the alpha is in getting early feedback from maintainers of re-usable modules we don't gave access to (because they are in private repositories), but the Terraform side of this is definitely still on our radar and likely to go in during a beta, assuming what we discussed above still seems approximately right (we can make small tweaks before final, of course).

Looks like there is a new, and pretty stable beta release of tf 0.12. Probably at a good place to start testing whether any changes might be needed to terragrunt itself.

We didn't get a chance to put in terragrunt variables migration yet because we wanted to focus on the main config language migration, but I hope we can still include something for this before rc1 if we have a specific format to target. Could also do it in a minor release later if necessary, though.

Do we know the minimum changes needed to user's terraform.tfvars files, and whether those changes will require any updates to terragrunt itself?

Hello again! :wave: It's been a while.
We're feeling close to final 0.12 release now, so I'm pondering what's best to do with what we discussed in this issue.
One option would be to work on upgrade rules for the terraform 0.12upgrade command that can transform a terragrunt pseudo-variable in a .tfvars file into one of the forms we discussed above, though if we do that before Terragrunt supports it I expect it will be confusing and may also impose constraints on a future implementation of loading that format in Terragrunt, since it would need to work with what Terraform generates even though we have no implementation to currently test it against.
A more straightforward change we could try instead is to detect if at least one .tfvars file contains a terragrunt definition that doesn't have a corresponding variable "terragrunt" block in configuration, and just produce a new terragrunt.tf file containing the following content:
variable "terragrunt" {
  type        = any
  description = "Configuration for terragrunt."
}

This should allow Terraform to accept and ignore a terragrunt definition in a .tfvars file without requiring significant changes to it, but it does still leave us with the problem that terraform 0.12upgrade is likely to make the terragrunt object not parseable by Terragrunt by escaping ${ sequences in the strings.
I think the most straightforward answer would be if Terragrunt were able to be configured to read its settings in the current format from some file other than terraform.tfvars that Terraform won't try to read by default, and then we could potentially make the upgrade codepath write the existing setting out verbatim (preserving the HCL 1 syntax using HCL 1's printer) into whatever other file makes sense, to create a short-term compromise: Terraform and Terragrunt look in separate files as I was proposing here, but we don't need to block on defining a whole new file format for the separate file.
We should still have at least a few weeks left before final to see if we can figure something out here. My preference would be to make a relatively small, straightforward change now, to minimize risk on both sides, and then allow Terraform's configuration format to evolve as appropriate separately later as you see fit. But mainly I want to be pragmatic here and find any reasonable way that'll allow Terragrunt users to migrate as gracefully as possible to Terraform 0.12. If you have other ideas, please let me know!

@apparentlymart Thanks for remembering us and checking in again! I think your idea of moving the Terragrunt config to a separate file, but in its old format, is a good one. That simplifies the upgrades we'll initially have to do to Terragrunt to support Terraform 0.12. I think a terragrunt.hcl file is probably a reasonable target?

Sounds reasonable to me! So, to make sure we're saying the same thing:
The terraform 0.12upgrade tool would look in all of the .tfvars files in the target directory (or maybe just terraform.tfvars?) and check for a definition for a variable named terragrunt. If that variable is not also declared using a variable "terragrunt" block (unlikely, but possible) then it will generate a file called terragrunt.hcl in the same directory and use the HCL 1 printer to print out as accurately as possible the string terragrunt =  followed by the AST that was assigned to the variable in the vars file.
If that sounds reasonable, I'll note that in our list of remaining tasks before final release and then, unless some other big issue crops up that we need to prioritize, we'll get that done for the final release.

Yes, I think that sounds right, but one issue just occurred to me... One of the key benefits of piggybacking on terraform.tfvars was that you could also define the values to set for the variables in your module, and Terraform would pick those up automatically. If we move the Terragrunt configuration to terragrunt.hcl, then you'll either need two files in every folder (and file sprawl is already a pain), or we'd need to implement extra logic to take the variables in terragrunt.hcl and forward them to Terraform (e.g., via -var).
In the ideas we wrote up way above, this meant moving those variables into an inputs { ... } block. But that brings us back to rewriting the code... Which isn't necessarily all that simple, right?

My instinct is to keep separate things separate: variables are for Terraform, and Terragrunt's configuration is for Terragrunt. Having one fewer file in the directory doesn't seem like a huge payoff for the complexity of having a single file be parsed in two completing ways by two pieces of software. :thinking:
Could we move forward with the proposed compromise in the short term, accepting that it means two files where there was only one before, and then look at ways for Terragrunt to forward variables from the terragrunt.hcl into Terraform itself as a separate step? I feel like once we have decoupled these two concerns it'll be much easier to make gradual incremental usability improvements on both sides, rather than us having to do heavy coordination like this every time.

Two files rather than one is no big deal. Four hundred files rather than two hundred files is. As the project scales up, doubling the number of files becomes a serious problem. I think it would be a significant step backwards to do that from a usability / maintenance perspective. Willing to hear other opinions from the community though.

I'd also be interested to learn more about this! I've not typically experienced number of files (specifically when split over many separate directories, not all in one directory) causing a problem; my concern is usually more about total size of all of the files summed together, which I think wouldn't be significantly larger here aside from some additional filesystem metadata.
With that said, if we can find a different way to go here which has similar effort to what we've been discussing so far then I'm all for it. (Keeping in mind that the alternative is Terragrunt not working with Terraform 0.12 at all, which seems worse.)

It's the same amount of data, 1 file or 2. No difference in my opinion. If the number of files is an issue for others, one option would be to switch to a single file or single directory of files that terragrunt reads, which contains the terragrunt config for the entire project.

Are there any current work-arounds to get Terragrunt working with Terraform 0.12? TF 0.12 is in rc1 right now (something I have been waiting months for), only to find that it doesn't work with Terragrunt yet.

@brikis98, @apparentlymart

One of the key benefits of piggybacking on terraform.tfvars was that you could also define the values to set for the variables in your module

I already use separate files for terragrunt config and terrraform vars like so:

the terraform.tfvars file only contains the terragrunt = {...} block
while the component.tfvars file contains the module's variables and is called with

      optional_var_files = [
        "${get_tfvars_dir()}/component.tfvars"
      ]

So going from a terraform.tfvars file to a terragrunt.hcl file is really straight forward in my case.
Althoug i want to make sure that my component.tfvars file as well as shared files in the project tree are not an issue for the terraform 0.12upgrade tool.

terraform 0.12upgrade works on only one directory at a time and is primarily concerned with upgrading .tf files; we intend for it to also do some processing of .tfvars files in the same directory as a convenience, but since .tfvars files can be in any directory in practice it can't possibly find all of the .tfvars files in your setup automatically. In situations where .tfvars files are kept somewhere other than the root module directory some additional work would be required to also "upgrade" the .tfvars files, separately from the configuration files that they are used with.
In practice the .tfvars upgrade rules are pretty simple -- they just deal with the fact that ${ now needs to be escaped to $${ to avoid being understood as a template interpolation -- so I expect most .tfvars existing (non-Terragrunt) files aren't using such sequences and won't need any help at all. You may find in your case that just renaming the terraform.tfvars file to terragrunt.hcl is sufficient, assuming Terragrunt were updated to know to read that file.

I am quite new to terragrunt and found a reference to an old .terragrunt configuration file name. Would moving back to old .terragrunt config files until the official support for 0.12 is released be a feasible option?

+20 on getting Terraform v.12 support for Terragrunt!

What's the plan? TF 0.12.0 was released today!

Even we are eagerly waiting for 0.12 support. 0.12 seems such a helpful release, migrating there asap would be awesome. Even https://github.com/gruntwork-io/terragrunt/issues/418 which is dependent on migrating to 0.12 would be super awesome.
Hoping we get a release soon :)

Sorry, this was closed, and referred to from #709 as the "solution", but... after reading the whole thread, I'm confused as to what we actually are expected to do. :/ I'm happy to share my present configuration. Are we supposed to go with the two-file solution? terragrunt.hcl? Thanks!

To put some weight behind a direction (single file vs multiple) - we do use multiple files at my company - but we go even further in that we do not commit any terragrunt specific files to our modules. We automatically add terragrunt config files/backend during the CICD process. The reason for not committing the tg config was so that we could use the same module code with different backends.
BUT, one of the reasons we automatically build the terragrunt files in tooling is that multiple files is definitely a complaint we heard from our teams, which is compounded when you have multiple environments and regions.
Maybe we can start with a separate file and find a way to re-integrate into a single file later? Folks are chomping at the bit to use tf 0.12 and that desire probably outweighs the overhead management of multiple files right now.

It seems as if we're at the point where Jim (@brikis98 ) still has reservations over assenting to the only proposed, agreeable to many, design+migration path with positive feedback:

The config lives in a separate .hcl file. The default name should be terragrunt.hcl.
The variables that used to be listed directly in terraform.tfvars now go in a variables = { ... } block.

To the casual observer, the above doesn't sound like a lot of work, but (IMHO)
it hides a non-trivial amount of work.
To me, it would appear to involve re-working terragrunt's existing HCL interpolation parser to use HCL2.
If I may, I'd like to offer one layer of simplification, which might provide a faster migration path to 0.12:
Can we name the file generated by the terraform 0.12upgrade tool to include the HCL version?

terragrunt.hcl1 rather than just .hcl

With a slight change to terragrunt's code to look for this new file, it would allow the current parser and interpolation hacks in terragrunt to be used.
terragrunt could someday include its own terragrunt hcl2-upgrade function once HCL2 support is added.
The community will reserve the terragrunt.hcl2 name, and users could expect that someday terragrunt.hcl2 (&&|| terragrunt.hcl) will be valid for HCL2 when that feature is available.

Happy with the two files approach, as some one who only uses one terraform.tfvars file including terragrunt and terraform configuration.
One feature request that I'd like to see to make a migration less painful, is for terragrunt looking through parent folders until it matches a terragrunt.hcl file rather than requiring one in the same folder.
This way, rather than needing to dupe the terragrunt.hcl files on migration throughout all configurations, I'd be able to define one terragrunt.hcl file at the right parent directory level for each module.
So allowing something like:
└── app
    ├── A
    │   └── terraform.tfvars
    ├── B
    │   └── terraform.tfvars
    ├── C
    │   └── terraform.tfvars
    │   └── terragrunt.hcl
    └── terragrunt.hcl

With that, it'd save me a lot of time in tedious configuration copying and allow for a one change update of module references for multiple deployments.

Another thing to consider is that we usually have global variables for multiple environments, for example, like so:
|---- infrastructure
       |
       |---- production
       |          |---- ecs-cluster
       |          |          |- terraform.tfvars
       |          |---- ecs-service
       |                     |- terraform.tfvars
       |---- staging
       |          |---- ecs-cluster
       |          |          |- terraform.tfvars
       |          |---- ecs-service
       |                     |- terraform.tfvars
       |
       |--- common.tfvars

And common.tfvars contains vpc_name variable, for example, which is defined and used by ecs-cluster, but not defined in ecs-service, and terraform will complain with "Warning: Value for undeclared variable"

Hi all, there is understandably a lot of interest in this GitHub issue and @brikis98 is on vacation this week so I wanted to post a quick update:
@brikis98 has been actively work on changes using the HCL2 library to upgrade Terragrunt so it can read from a terragrunt.hcl file. The file will use HCL2 syntax, and, thanks to the fact that the HCL2 parser natively supports functions and variables, will support first-class expressions everywhere (i.e., use Terragrunt helpers anywhere in the file).
Based on what we know today, it does not seem to be a large change, so our good-faith could-be-wrong estimate is that we will complete this work by the end of next week (June 7). Thanks for your patience.

While we are here, would it be possible to allow to specify path to terraform in terragrunt.hcl to ease migration? So that terraform version can be set on per-module basis, or globally for a project. Should fix #265 as well.

Update: I've submitted a PR that adds Terraform 0.12 support. I think it's good to go, but a few tests are failing, possibly due to a bug in Terraform 0.12. Follow the PR for details.

Alright, #731 is merged. Please give https://github.com/gruntwork-io/terragrunt/releases/tag/v0.19.0 a try (the new binaries should show up in a few minutes)!

@brikis98
Thanks to make terragrunt ready for terraform 0.12
But I got error:
/apps # terragrunt version
[terragrunt] [/apps] 2019/06/11 06:20:31 Running command: terraform --version
[terragrunt] 2019/06/11 06:20:32 Reading Terragrunt config file at /apps/terragrunt.hcl
[terragrunt] 2019/06/11 06:20:32 Error reading file at path /apps/terragrunt.hcl: open /apps/terragrunt.hcl: no such file or directory
[terragrunt] 2019/06/11 06:20:32 Unable to determine underlying exit code, so Terragrunt will exit with error code 1

/apps # terraform --version
Terraform v0.12.1

/apps # terragrunt init
[terragrunt] [/apps] 2019/06/11 06:20:44 Running command: terraform --version
[terragrunt] 2019/06/11 06:20:44 Reading Terragrunt config file at /apps/terragrunt.hcl
[terragrunt] 2019/06/11 06:20:44 Error reading file at path /apps/terragrunt.hcl: open /apps/terragrunt.hcl: no such file or directory
[terragrunt] 2019/06/11 06:20:44 Unable to determine underlying exit code, so Terragrunt will exit with error code 1
/apps #

@ozbillwang Glad to hear you got it sorted!

@brikis98
And I can confirm now, I run on a complex terraform stack with 7 terraform modules, the part to do 0.12upgrade in terraform takes time to make it work, but the part to run terragrunt after 0.12upgrade, it works perfect.

While declaring an empty remote state in a way:
terraform {
  # The configuration for this backend will be filled in by Terragrunt
  backend "gcs" {}
}

I'm getting:
Error: Missing key/value separator

  on terraform.tfvars line 2:
   1:
   2:   terraform {

Expected an equals sign ("=") to mark the beginning of the attribute value.

terragrunt version v0.19.2
Terraform v0.12.2
+ provider.google v2.6.0

on terraform.tfvars line 2:

You still have a terraform.tfvars file hanging around?

The only file that's called terraform.tfvars is included here:
"-var-file=${get_terragrunt_dir()}/../tfvars/terraform.tfvars",

And does it have a terraform { ... } block starting on line 2?

No.

That error is from Terraform itself, not Terragrunt...
Are you passing terragrunt.hcl as a var file to Terraform somewhere? Check all your extra_arguments declarations.

Nevermind, I had leftover environment variables set... Excuse my stupidity.

No prob! Thx for following up.

Hi @brikis98, I cant find any guide on below link... please help
https://github.com/gruntwork-io/terragrunt/blob/master/_docs/migration_guides/upgrading_to_terragrunt_0.19.x.md

Try https://github.com/gruntwork-io/terragrunt/blob/v0.19.0/_docs/migration_guides/upgrading_to_terragrunt_0.19.x.md


bdos: revert previous fix. it broke listing the directory and probably more

Sorry. Guess I was too exited. I'll look into it more thoroughly.

Merged. Cheers.
But the MAME tests passed, so it can't have been that broken. They don't test directory listing but they do test file loading and saving.


109 updating documentation for v1.0

I added some content to the docs:

Home/index.md
Explanation:

Levels of schemas,
Schemas and Readers (just a place holder)

How-to Guids:

Install this Plugin
Use this Plugin (just a place holder)
Develop this Plugin (just a place holder)

Contact (analg to the simulations plugin from Area C)

Todos:

[ ] review this new documents
[ ] add content to "Schemas and Readers"
[ ] add content to "Use this Plugin"
[ ] add content to "Develop this Plugin"

Questions:

Do we put something into "References" and "Tutorials"?
The documentation of NOMAD-Materials-Processing is quite analog to this one. We should consicer pushing the general stuff to the central NOMAD docs Domain-specific Examples!?

explaination: XRD @ka-sarthak
how-tos same as in materials processing

as is : XRD @ka-sarthak (drag and drop a file), show that it gets parsed automatically
inherit: yaml plus sth extra in XRD @ka-sarthak
python: @hampusnasstrom

contact: topics remove @budschi


Requesting USI Life Support be added to CKAN

This mod doesn't seem to be in the database yet.
https://github.com/BobPalmer is the author of this mod
https://github.com/BobPalmer/USI-LS/releases/tag/0.1.0 is the mod in question.
I'm just requesting this be added because the Wiki told me it was acceptable to do so. :)

See https://github.com/KSP-CKAN/NetKAN/pull/1021 we're waiting for a first stable release so our bot can index it.

Prerelease indexed into CKAN manually now.

Thanks!


REQUEST : Add a way to send command

Make a way to send message or moderate the chat from our client

Hi,
This requires a fresh msToken signature for each request. Sorry, we do not support authenticated endpoints.


conditionally render in cell editor is not working with ag-grid autocomplete

conditionally render in cell editor is not working with ag-grid autocomplete
In the below example I am using two types of cell editor select & normal text editor.
Select editor is rendered when (params.data.contextValueTypeName === "Select") else normal text editor is rendered.
But when I am using AutocompleteSelectCellEditor in place of select editor it is giving some error.
columnDefs: [
        {
          headerName: "Value",
          field: "valueField",
          editable: true,
          cellEditorSelector: params => {
            if (params.data.contextValueTypeName === "Select") {
              return {
                component: "agRichSelectCellEditor",
                params: {
                  values: params.data.listItems
                }
              };
            }
            return null;
          }
]

Hi there !
If you want to reference ag-grid components that way (with string name) according to ag-grid documentation you have to register it into your gridOptions first.
Something like so:
import {AutocompleteSelectCellEditor} from 'ag-grid-autocomplete-editor';
import {Grid} from 'ag-grid-community';

const gridOptions = {
    columnDefs: columnDefs,
    rowData: rowDatas,
    suppressScrollOnNewData: false,
    components: {
      // Register component into agGrid
      agGridAutocompleteSelectEditor: AutocompleteSelectCellEditor,
    }
};
...

const columnDefs = [
    {
        headerName: "Already present data selector",
        field: "make",
        cellEditorSelector: (params) => {
              return {
                // Use component by string name
                component: "agGridAutocompleteSelectEditor",
                params: {
                  required: true,
                  selectData: selectData,
                  placeholder: 'Select an option',
                }
              };
        },
        ...
    },
];

I made a little example for your use case here.


fix(docs): reported issue with example packages

This updates the exposed docs as follows:

Fixes the configuration example for two articles (per reported issue)
Hides the article tagging cookbook recipe (pending restoration of function from plugin)
Adds test for the two impacted docs
Includes new generated SDK API docs from the make html bits.

Fixes #450

Failure to reach a plugin in the generator tests should not prevent docs updates.


Remove Objective Events from the engine

Description
Removes Objective events from the engine.
This simplifies dealing with the engine. Instead of listening for objective events, objectives are returned from the engine api.
The wallet is also now an EventEmitter with one event ObjectiveProposed.  This event fires when the wallet receives a new proposed objective from a participant.  wallet.approveObjectives can then be used to approve the objective.
Currently there is still a channel-updated event. I think this can be deprecated as well but I would like to tackle this in a separate PR.

Checklist:
Code quality

[x] I have written clear commit messages
[ ] I have performed a self-review of my own code
[x] This change does not have an unduly wide scope
[x] I have separated logic changes from refactor changes (formatting, renames, etc.)
[ ] I have commented my code wherever necessary (can be 0)
[ ] I have added tests that prove my fix is effective or that my feature works, if necessary

Project management

[x] I have applied the appropriate labels
[ ] I have linked to all relevant issues (can be 0)
[ ] I have linked to all dependent issues (can be 0)
[x] I have assigned myself to this PR
[ ] I have chosen the appropriate pipeline on zenhub for the linked issue

Re. 18f62e8: what motivated this change?

Meta comment: Do these comments survive a rebase?

Re. 18f62e8: what motivated this change?

Meta comment: Do these comments survive a rebase?

It does on my screen!

Sorry, my comment was misleading!
I haven't done a rebase since that comment I was just curious what would happen if I do.

Sorry, my comment was misleading!
I haven't done a rebase since that comment I was just curious what would happen if I do.

... Do you want to try? 😄


buf lint trying to lint file that doesn't exist

example run: https://github.com/harmony-development/protocol/runs/4670337736?check_suite_focus=true
with a configuration like:
  lint-protos-staging:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses<EMAIL_ADDRESS>      - uses: bufbuild/buf-lint-action@v1
        with:
          input: 'staging'

and a file structure like
 .
├──  CMakeLists.txt
├──  FAQ.md
├──  GUIDELINES.md
├──  LICENSE
├──  README.md
├──  shell.nix
├──  stable
│  ├──  auth
│  │  └──  v1
│  │     └──  auth.proto
│  ├──  batch
│  │  └──  v1
│  │     └──  batch.proto
│  ├──  buf.yaml -> ../.buf.yaml
│  ├──  chat
│  │  └──  v1
│  │     ├──  channels.proto
│  │     ├──  chat.proto
│  │     ├──  guilds.proto
│  │     ├──  messages.proto
│  │     ├──  Permissions.md
│  │     ├──  permissions.proto
│  │     └──  stream.proto
│  ├──  CMakeLists.txt
│  ├──  emote
│  │  └──  v1
│  │     ├──  emote.proto
│  │     ├──  stream.proto
│  │     └──  types.proto
│  ├──  harmonytypes
│  │  └──  v1
│  │     └──  types.proto
│  ├──  mediaproxy
│  │  └──  v1
│  │     └──  mediaproxy.proto
│  ├──  name-resolution
│  │  ├──  example
│  │  │  ├──  go.mod
│  │  │  └──  main.go
│  │  └──  name-resolution.md
│  ├──  profile
│  │  └──  v1
│  │     ├──  appdata.proto
│  │     ├──  profile.proto
│  │     ├──  stream.proto
│  │     └──  types.proto
│  ├──  rest
│  │  └──  rest.md
│  └──  sync
│     └──  v1
│        └──  sync.proto
└──  staging
   ├──  buf.yaml -> ../.buf.yaml
   ├──  CMakeLists.txt
   └──  voice
      └──  v1
         └──  voice.proto

buf attempts to lint a file from stable, but fails because it's not in the staging directory

Hi, I took a look at this by cloning the source repo and running buf lint staging locally against the head of main. It appears that the file in staging has a dependency on the file in stable, but since you are only targeting the staging directory as your input, it is unable to build because the dependency is not found. There are a few options that you can try:

Running lint against the entire repository but use the ignore field in the lint configuration for your module to exclude the stable directory (docs available here: https://docs.buf.build/lint/configuration#ignore). Note that in this case, your module file will be the one you have defined here https://github.com/harmony-development/protocol/blob/main/.buf.yaml and they symlinks would be removed from the subdirectories.
Separating staging and stable into two different modules. However, they would need individually defined buf.yaml configurations (instead of the symlinked configs), and a dependency would be declared on the other. You can then use a workspace file to join these dependencies. Documentation for workspaces and the configuration files are available here:

https://docs.buf.build/reference/workspaces
https://docs.buf.build/configuration/v1/buf-yaml
The tour also walks through an example of setting up a workspace with two modules that depend on one another.

Hope this helps and let me know if you have any other questions!

ah, i see :+1:
consider this a bug report for an unhelpful error message then, since "file not found" without further context makes it sound like buf is trying to lint something that doesn't exist instead of an import not being found

"file not found" without further context

Agreed. Would something like voice/v1/voice.proto:3: import harmonytypes/v1/types.proto not found provide sufficient context?

yes, that would be a big improvement from the status quo

Closing as the issue above has been merged.


Persistent bug during dp-bert-large-pretrain example

Description
I'm unable to run the trainium-inferentia BERT pretrain model. Following error is showing up during building:
Traceback (most recent call last):   File "/home/ec2-user/.local/bin/torchx", line 8, in <module>     sys.exit(main())   File "/home/ec2-user/.local/lib/python3.8/site-packages/torchx/cli/main.py", line 116, in main     run_main(get_sub_cmds(), argv)   File "/home/ec2-user/.local/lib/python3.8/site-packages/torchx/cli/main.py", line 112, in run_main     args.func(args)   File "/home/ec2-user/.local/lib/python3.8/site-packages/torchx/cli/cmd_run.py", line 248, in run     self._run(runner, args)   File "/home/ec2-user/.local/lib/python3.8/site-packages/torchx/cli/cmd_run.py", line 208, in _run     app_handle = runner.run_component(   File "/home/ec2-user/.local/lib/python3.8/site-packages/torchx/runner/api.py", line 186, in run_component     return self.schedule(dryrun_info)   File "/home/ec2-user/.local/lib/python3.8/site-packages/torchx/runner/api.py", line 278, in schedule     app_id = sched.schedule(dryrun_info)   File "/home/ec2-user/.local/lib/python3.8/site-packages/torchx/schedulers/kubernetes_scheduler.py", line 593, in schedule     resp = self._custom_objects_api().create_namespaced_custom_object(   File "/home/ec2-user/.local/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py", line 231, in create_namespaced_custom_object     return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501   File "/home/ec2-user/.local/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py", line 354, in create_namespaced_custom_object_with_http_info     return self.api_client.call_api(   File "/home/ec2-user/.local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api     return self.__call_api(resource_path, method,   File "/home/ec2-user/.local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api     response_data = self.request(   File "/home/ec2-user/.local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 391, in request     return self.rest_client.POST(url,   File "/home/ec2-user/.local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 279, in POST     return self.request("POST", url,   File "/home/ec2-user/.local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 238, in request     raise ApiException(http_resp=r) kubernetes.client.exceptions.ApiException: (400) Reason: Bad Request HTTP response headers: HTTPHeaderDict({'Audit-Id': '9ea0bf3e-2327-45ae-aefa-0965f38155ff', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '344539cc-94a8-443b-82c3-e6ffd6feb173', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'bb6f7b9d-abeb-49b1-bcda-ff2bc8c180bf', 'Date': 'Tue, 23 Jan 2024 22:43:20 GMT', 'Content-Length': '232'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"admission webhook \"validatejob.volcano.sh\" denied the request: unable to find job queue: queues.scheduling.volcano.sh \"test\" not found;","code":400}
EKS Data blueprint was provided from [https://awslabs.github.io/data-on-eks/docs/blueprints/ai-ml/trainium](EKS Data url)
I re-initialized the project several times in both Cloud9 and in my local system. Both with the same result.
I re-attempt the terraform ./install.sh file.
Versions

ai-ml/trainium-inferentia: 39e790ce0d4e45979d1374a86b2030e55a838441

Terraform version:
Terraform v1.5.5 on linux_amd64

Provider version(s):
Terraform v1.5.5 on linux_amd64

Reproduction Code [Required]
cd ai-ml/trainium-inferentia/examples/dp-bert-large-pretrain
chomd +x 2-bert-pretrain-precompile.sh
./2-bert-pretrain-precompile.sh
Workspace used: Cloud9 following along this Workshop [https://www.eksworkshop.com/docs/introduction/setup/your-account/]( EKS Workshop url)
List steps in order that led up to the issue you encountered
`cd data-on-eks/ai-ml/trainium/ && chmod +x install.sh
./install.sh
cd ai-ml/trainium-inferentia/examples/dp-bert-large-pretrain
chomd +x 1-bert-pretrain-build-image.sh
./1-bert-pretrain-build-image.sh
kubectl exec -i -t -n default aws-cli-cmd-shell -c app -- sh -c "clear; (bash || ash || sh)"
yum install tar
cd /data
aws s3 cp s3://neuron-s3/training_datasets/bert_pretrain_wikicorpus_tokenized_hdf5/bert_pretrain_wikicorpus_tokenized_hdf5_seqlen128.tar . --no-sign-request
chmod 744 bert_pretrain_wikicorpus_tokenized_hdf5_seqlen128.tar
tar xvf bert_pretrain_wikicorpus_tokenized_hdf5_seqlen128.tar
`
Expected behavior
Pretrain Bert Model successfully built
Actual behavior
Traceback (most recent call last):   File "/home/ec2-user/.local/bin/torchx", line 8, in <module>     sys.exit(main())   File "/home/ec2-user/.local/lib/python3.8/site-packages/torchx/cli/main.py", line 116, in main     run_main(get_sub_cmds(), argv)   File "/home/ec2-user/.local/lib/python3.8/site-packages/torchx/cli/main.py", line 112, in run_main     args.func(args)   File "/home/ec2-user/.local/lib/python3.8/site-packages/torchx/cli/cmd_run.py", line 248, in run     self._run(runner, args)   File "/home/ec2-user/.local/lib/python3.8/site-packages/torchx/cli/cmd_run.py", line 208, in _run     app_handle = runner.run_component(   File "/home/ec2-user/.local/lib/python3.8/site-packages/torchx/runner/api.py", line 186, in run_component     return self.schedule(dryrun_info)   File "/home/ec2-user/.local/lib/python3.8/site-packages/torchx/runner/api.py", line 278, in schedule     app_id = sched.schedule(dryrun_info)   File "/home/ec2-user/.local/lib/python3.8/site-packages/torchx/schedulers/kubernetes_scheduler.py", line 593, in schedule     resp = self._custom_objects_api().create_namespaced_custom_object(   File "/home/ec2-user/.local/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py", line 231, in create_namespaced_custom_object     return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501   File "/home/ec2-user/.local/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py", line 354, in create_namespaced_custom_object_with_http_info     return self.api_client.call_api(   File "/home/ec2-user/.local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api     return self.__call_api(resource_path, method,   File "/home/ec2-user/.local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api     response_data = self.request(   File "/home/ec2-user/.local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 391, in request     return self.rest_client.POST(url,   File "/home/ec2-user/.local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 279, in POST     return self.request("POST", url,   File "/home/ec2-user/.local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 238, in request     raise ApiException(http_resp=r) kubernetes.client.exceptions.ApiException: (400) Reason: Bad Request HTTP response headers: HTTPHeaderDict({'Audit-Id': '9ea0bf3e-2327-45ae-aefa-0965f38155ff', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '344539cc-94a8-443b-82c3-e6ffd6feb173', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'bb6f7b9d-abeb-49b1-bcda-ff2bc8c180bf', 'Date': 'Tue, 23 Jan 2024 22:43:20 GMT', 'Content-Length': '232'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"admission webhook \"validatejob.volcano.sh\" denied the request: unable to find job queue: queues.scheduling.volcano.sh \"test\" not found;","code":400}
Terminal Output Screenshot(s)

Additional context
Trainium on EKS blueprint

Thanks for raising the issue. I will try this blueprint and update the same to the issue.

 "Failure","message":"admission webhook \"validatejob.volcano.sh\" denied the request: unable to find job queue: queues.scheduling.volcano.sh \"test\" not found;","code":400}
Just noticed the above error indicates the job queue is missing for Volcano.  Try to kubectl apply the below manifest that will create namespace and the Volcano queue and try to run the shell script(2-bert-pretrain-precompile.sh) again.
---
apiVersion: v1
kind: Namespace
metadata:
  name: test

# Volcano dedicated queue for ml-team-a
---
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: test
spec:
  reclaimable: false
  weight: 1

We can update the blueprint if this works

Hi @vara-bonthu ,
I successfully added and applied the manifest yaml file.
The "Failure","message":"admission webhook \"validatejob.volcano.sh\" denied the request: unable to find job queue: queues.scheduling.volcano.sh \"test\" not found;","code":400} error is no longer showing up, but I'm still not able to run the bert-compile pods.
I'm attaching the screenshots to show my results:

It seems you've made good progress. The BERT large distributed training blueprint is utilizing Managed Node Groups, so you'll need to set the minimum and desired values to 2. These values can be updated in the variables.tf file. Here are the specific lines where you can make these changes:
Nodes Minimum Value
Nodes Desired Value
After making these adjustments , please run terraform apply. This will provision two nodes of trn1.32xlarge instances. Ensure that your account has access to these nodes.
Upon completion, you should observe the pending pods transitioning to the running state.
I've noticed some gaps in the documentation that need updating. Thanks for validating and I appreciate a PR for these missing steps. Thank you!

@Gall-oDrone This blueprint has been recently updated with the fixes. Checkout the latest PR here https://github.com/awslabs/data-on-eks/pull/435


Add a fixit for adding "jsx: true" to the tsconfig

Search Terms
fixit quick fix jsx flag tsconfig
Suggestion
Hitting "Quick Fix" in this error could offer to add the flag for you

Use Cases
Anyone starting a reactproject
Checklist
My suggestion meets these guidelines:

[x] This wouldn't be a breaking change in existing TypeScript/JavaScript code
[x] This wouldn't change the runtime behavior of existing JavaScript code
[x] This could be implemented without emitting different JS based on the types of the expressions
[x] This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, etc.)
[x] This feature would agree with the rest of TypeScript's Design Goals.

I would love to take this if possible.


WPP_GLOBAL_Control clarification

While looking at the code I noticed following snippet, extracted from preprocessed ParaNdis_Common.i file:
static __inline void
DumpMac(int dbg_level, const char* calling_function, const char* header_str, UCHAR* mac)
{
      __annotation( L"TMF:", L"a95c641b-f553-3765-860d-0d03cf3c0e75 netkvm.sys // SRC=" L"ParaNdis_Common.cpp" L" MJ= MN=", L"#typev " L"ParaNdis_Common_cpp601" L" 28 \"%0 %!FUNC![%10!s!] - %11!s!: %12!02x!-%13!02x!-%14!02x!-%15!02x!-%16!02x!-%17!02x!\" //   Flags=TRACE_DRIVER LEVEL=dbg_level" , L"{", L"calling_function, ItemString -- 10" , L"header_str, ItemString -- 11" , L"mac[0], ItemLong -- 12" , L"mac[1], ItemLong -- 13" , L"mac[2], ItemLong -- 14" , L"mac[3], ItemLong -- 15" , L"mac[4], ItemLong -- 16" , L"mac[5], ItemLong -- 17" , L"}"), (( (WPP_GLOBAL_Control != (WPP_PROJECT_CONTROL_BLOCK*)&WPP_GLOBAL_Control) && (dbg_level <= virtioDebugLevel) ? (void)0, WPP_SF_ssDDDDDD( ((WPP_GLOBAL_Control[((WPP_BIT_TRACE_DRIVER) >> 16)].Control).Logger), 28, WPP_a95c641bf5533765860d0d03cf3c0e75_Traceguids+0, calling_function, header_str, mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]), 1 : 0 )) ;
#line 604 "C:\\Users\\User\\source\\repos\\kvm-guest-drivers-windows\\NetKVM\\Common\\ParaNdis_Common.cpp"

}

The bit that I'm interested in is WPP_GLOBAL_Control != (WPP_PROJECT_CONTROL_BLOCK*)&WPP_GLOBAL_Control condition. I would like to understand it a bit more, it's likely I'm missing something here, as it seems to me it should be the same as WPP_GLOBAL_Control != WPP_GLOBAL_Control.

Original code before expansion:
https://github.com/virtio-win/kvm-guest-drivers-windows/blob/72f495033e6710d6459f9be6c706576caedda876/NetKVM/Common/ParaNdis_Common.cpp#L599-L604

Is there something specific to netkvm or virtio-win? Or the same happens also with other drivers/not only drivers that use WPP?

I can't quite tell yet, I'm not really familiar with developing drivers on Windows and when filing I was not aware WPP is part provided by MS. That said I have not yet worked out where does the condition get injected from.


Error in README

The end of the readme is in a code block. We need to make sure that only the sample output is in the code block.

see branch titled "sprint-1-hotfixes"


Sonoff POW R2

Hi, FIRST OF ALL: thanks for this implementation... no need for firmware replacement... VERY NICE!
Second, how can I help to add support for Sonoff POW R2?
I'd like to read the consumption data on the Home app.
Also, how about create a wiki with curl commands from the API so we can map all RAW data and protocol from sonoff devices?

Hi @luizjunior, it would be great if you can help on adding support for the Sonoff POW R2! This project is open source and have a relaxed license that you are free to modify the code and share your work. You can look into the source code, it's written in Javascript which is fairly easy to understand and write. To contribute code back to this project, please fork the repository, make the necessary changes and then send me a pull request.
To intercept the API request / response data, please refer to the README on the original homebridge-ewelink project, which have a section showing how to observe the data with Charles.

anyone gotten the POW R2 to work?
i can see that homebridge gets the info about power in the log
[5/11/2019, 1:32:20 PM] [eWeLink] Error updating non-exist accessory with deviceId [XXXXXXX].
[5/11/2019, 1:34:20 PM] [eWeLink] WebSocket messge received:  {"action":"update","deviceid":"10007c6ca5","apikey":"XXXXXXXXXXXXXXXX","userAgent":"device","ts":0,"params":{"power":"7.28","voltage":"235.56","switch":"on"},"from":"device"}

+1 for this feature. The most elegant solution would be to have power consumption in Watt as a sensor value, but looking at the full list of Apple HomeKit Characteristic Types (https://developer.apple.com/documentation/homekit/hmcharacteristic/characteristic_types), it seems that no "power consumption" characteristic exists yet. There are temperature, humidity, motion and various "on/off" states, but none seem suitable for power consumption.
For my use case (seeing if the washing machine is still running or has finished), it would already be a big step forward if I could show something like an "Outlet in Use" status for the device when power consumption is above a certain threshold. That way I can set up automations in the Apple Home App to notify me when the laundry is ready.
I can also see the power level coming in as events in the log, but not sure how to go about adding new characteristics to the switch in the plugin. Any thoughts?


UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4: ordinal not in range(128)

Hello,
We are facing an issue with powerdnsadmin when trying to open a domain.
Aug 16 08:15:25 ns1-server gunicorn[877634]: [2021-08-16 08:15:25,122] [app.py:1891] ERROR - Exception on /domain/
*hidden [GET]
Aug 16 08:15:25 ns1-server gunicorn[877634]: Traceback (most recent call last):
Aug 16 08:15:25 ns1-server gunicorn[877634]:   File "/var/www/html/pdns/powerdnsadmin/lib/utils.py", line 244, in pretty_domain_name
Aug 16 08:15:25 ns1-server gunicorn[877634]:     return value.encode().decode('idna')
Aug 16 08:15:25 ns1-server gunicorn[877634]:   File "/usr/lib/python3.8/encodings/idna.py", line 214, in decode
Aug 16 08:15:25 ns1-server gunicorn[877634]:     result.append(ToUnicode(label))
Aug 16 08:15:25 ns1-server gunicorn[877634]:   File "/usr/lib/python3.8/encodings/idna.py", line 125, in ToUnicode
Aug 16 08:15:25 ns1-server gunicorn[877634]:     return str(label, "ascii")
The issue occurs with the record "v=spf1 include:spf.protection.outlook.com –all".
+--------+-----------+--------------------------+-------+--------------------------------------------------------------------------+-------+------+-----------+------+----------+
| id     | domain_id | name                     | type  | content                                                                  | ttl   | prio | ordername | auth | disabled |
+--------+-----------+--------------------------+-------+--------------------------------------------------------------------------+-------+------+-----------+------+----------+
| 178332 |      2691 | hidden              | SOA   | hidden*. hostmaster.hidden*.<PHONE_NUMBER> 21600 7200 2419200 86400 | 28800 |    0 | NULL      |    1 |        0 |
| 178333 |      2691 | hidden              | NS    | hidden*                                                            | 28800 |    0 | NULL      |    1 |        0 |
| 178334 |      2691 | hidden              | NS    | hidden*                                                            | 28800 |    0 | NULL      |    1 |        0 |
| 178335 |      2691 | hidden              | TXT   | "v=spf1 include:spf.protection.outlook.com –all"                         | 28800 |    0 | NULL      |    1 |        0 |
+--------+-----------+--------------------------+-------+--------------------------------------------------------------------------+-------+------+-----------+------+----------+
If we remove id 178335 from the database, the issue is no longer present and the domain is showing up.
If we recreate the record again in powerdns-admin, the domain is again no longer able to open and shows an 500 error again.
Can you please have a check what is going on?

it seems that the creation of the txt record was made incorrect but i could not identify that there are special characters in the record.
I only saw it using "dig".
3600    IN      TXT     "v=spf1 include:spf.protection.outlook.com \226\128\147all"


EvalSum bug

Hello, I'm doing some tests with the python wrapper and I noticed a bug(?) with the EvalSum function with batch sizes 1024+.
In the following Euclidean distance, the sum returns wrong results (even negative numbers). Everything works fine with batch sizes 128, 256 and 512.
`   filename_a = "1000_1024_1_a.npy"
filename_b = "1000_1024_1_b.npy"
dataset_a = numpy.load(filename_a)
dataset_b = numpy.load(filename_b)

v1 = list(dataset_a[0])
v2 = list(dataset_b[0])

mult_depth = 1
scale_mod_size = 50
batch_size = len(v1)

parameters = openfhe.CCParamsCKKSRNS()
parameters.SetMultiplicativeDepth(mult_depth)
parameters.SetScalingModSize(scale_mod_size)
parameters.SetBatchSize(batch_size)
parameters.SetScalingTechnique(ScalingTechnique.FLEXIBLEAUTOEXT)

cc = openfhe.GenCryptoContext(parameters)
cc.Enable(openfhe.PKESchemeFeature.PKE)
cc.Enable(openfhe.PKESchemeFeature.KEYSWITCH)
cc.Enable(openfhe.PKESchemeFeature.LEVELEDSHE)
cc.Enable(openfhe.PKESchemeFeature.ADVANCEDSHE)

keys = cc.KeyGen()
cc.EvalMultKeyGen(keys.secretKey)
cc.EvalRotateKeyGen(keys.secretKey, [2 ** n for n in range(0, math.ceil(math.log2(batch_size)) + 1)])

print("keys generated")

# Encoding and encryption
# The operation is performed between encrypted v1 and plaintext v2
ptx1 = cc.MakeCKKSPackedPlaintext(v1)
ptx2 = cc.MakeCKKSPackedPlaintext(v2)

ctx1 = cc.Encrypt(keys.publicKey, ptx1)

# Component-wise difference: ctx - ptx
diff_ctx = cc.EvalSub(ctx1, ptx2)

# Component-wise square: (ctx - ptx)^2
square_ctx = cc.EvalSquare(diff_ctx)

# Sum of all the components
squared_ed_ctx = cc.EvalSum(square_ctx, batch_size)

res = cc.Decrypt(keys.secretKey, squared_ed_ctx)
res.SetLength(batch_size)

value = res.GetRealPackedValue()[0]

print(f"Plaintext value: {euclidean_distance(v1,v2)}")
print(f"HE value: {math.sqrt(value)}")`

`
def euclidean_distance(v1, v2):
v1 = np.array(v1)
v2 = np.array(v2)
ed = np.sqrt(np.sum(np.square(v1-v2)))
return ed`

However, setting multiplicative depth to 2 everything works fine. Does the EvalSum consume a level? Is it supposed to do so?
Files to reproduce the bug are attached.
1000_1024_1_.zip

This depends on the norm of the input values. If the sum is higher than roughly $2^{60}/2^{50}$ (for your cryptocontext parameters), then you will have an overflow and need an extra RNS limb (level) to support larger numbers. This issue has been discussed/answered multiple times in the Discourse forum. It does not look like a bug to me.

Actually, the inputs are vectors of 1024 randomly generated elements in [-1,1]. Even with the square, the sum could never reach $2^{50}$.

See my response at https://openfhe.discourse.group/t/evalsum-possible-bug/1533/4?u=ypolyakov It is not a bug. This behavior is by design.